Monitoring physiological conditions

ABSTRACT

A method for unobtrusive screening the well-being of a person using a plurality of sensing devices, including detecting the presence of a person; identifying and tracking the person; using sensing devices, which includes at least one camera, to unobtrusively sense one or more body parameters of the person; calculating and storing wellness parameters, which are derived from the sensed body parameter data; and automatically evaluating the well-being of the person based on current and prior semantic data including comparing the wellness parameters to previously determined wellness parameters for that person.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.11/751,645 filed May 22, 2007, which is incorporated herein by referencein its entirety.

Reference is made to commonly assigned U.S. patent application Ser. No.11/555,822, filed Nov. 2, 2006, entitled “An Integrated Display HavingMultiple Capture Devices”, by Kurtz et al, U.S. patent application Ser.No. 11/751,646 filed May 22, 2007, entitled “Establishing Baseline Datafor Physiological Monitoring System” by Gobeyn et al, U.S. patentapplication Ser. No. 11/751,648 filed May 22, 2007, entitled “CapturingData for Individual Physiological Monitoring” by Kurtz et al, U.S.patent application Ser. No. 11/751,652 filed May 22, 2007, entitled“Image Data Normalization for a Monitoring System” by Gobeyn et al, U.S.patent application Ser. No. 11/751,657 filed May 22, 2007, entitled“Inferring Wellness from Physiological Condition Data” by Gobeyn et aland U.S. patent application Ser. No. 11/751,660 filed May 22, 2007,entitled “Privacy Management for Well-Being Monitoring” by Kurtz et al,the disclosures of which are incorporated herein.

FIELD OF THE INVENTION

The present invention relates to monitoring the physiological conditionsof one or more individuals in an unobtrusive ongoing manner, by usingimages acquired by one or more digital capture devices.

BACKGROUND OF THE INVENTION

In many places worldwide, population demographics point persuasivelytowards a gradually emerging mismatch between the need for medical careamong aging populations, and the number of health care providersavailable to respond to the need. There are many proposals, fromincreased training of medical professionals, to telemedicine, andelectronic patient records, which may satisfy the needs to an extent.Telemedicine can fulfill an immediate need for consultation by aremotely located physician or specialist. Telemedicine could also beapplied to the monitoring of slowly evolving medical conditions, such aschronic wounds or infections. However, there is certainly room for amultiplicity of solutions.

As a parallel and interacting societal and cultural trend, the Internetis enabling families and individuals to have greater influence on theirown health and well being, as health and medical information has becomeincreasingly accessible. However, in many instances, significant timecan pass during which subtle physiological changes can be occurring toan individual without their awareness. Alternately, individuals can beaware of a more obvious physiological change, but lack any way toquantify and confirm the change. Thus, a system or device that candetect, quantify, and track physiological changes would have significantutility. Examples of such physiological conditions might include changesin nutrition, weight gain or loss, and emotional state. As a furthercapability, the aforementioned device could have the capability to alertan individual, family members, and their health care provider(s) todetected changes. The system could also interact with a database toscreen and tentatively identify relevant medical conditions.

As can be seen, a device or system with aforementioned attributes wouldhave considerable value. To realize this value, such a system or deviceshould be sufficiently inexpensive to be consumer accessible. The systemshould also have attributes, such as multi-functionality, autonomousoperation, ease of use, and perhaps portability, to have it function asa natural part of the consumer or home environment. It should operateunobtrusively, collecting useful data while reducing its interactionrequirements and maintaining user privacy. Preferably it can be usefulfor tracking a broad range of physiologic conditions (such as nutrition,weight, or posture, for example), some medical conditions, and havecosmetic applications as well. Likewise, the system should besufficiently flexible to function properly for different individuals(such as different family members), and be able to accommodate ethnic,seasonal, and cultural differences. Such a system might be expected tobe imaging based, but also accept other sensory inputs. A system withthese features would enable many individuals to address their health andwell-being issues more pro-actively.

There are prior art systems that consider some of the issues describedabove. As a first example, U.S. Pat. No. 5,437,278 by Wilk describes amedical diagnostic system that collects medical data from an imagingsystem or from a physiological measurement device. The Wilk '278 systemattempts to automate medical data collection and diagnosis. For example,video image data is collected, and these images and related imagingparameters are compared to a master database to facilitate medicaldiagnosis. The collected images are also compared to prior scannedpatient data to facilitate ongoing patient monitoring. Although it isstrongly implied, rather than explicitly stated, the Wilk '278 system istargeted for use in clinical environment. For one, the system of Wilk'278 is to be operated by a health care professional or an unskilledaide. Wilk '278 also anticipates that the imaging device can be a videocamera, an X-ray machine, an MRI scanner, or a CAT scanner. The systemis also intended to accept inputs from EEG and EKG machines and othermonitoring devices. As can be seen, Wilk '278 employs expensive medicalmachinery that is expected to be in a hospital or clinic, and not in ahome. Thus Wilk' 278 does not propose a system for monitoringphysiological conditions that is applicable to the home environment.

As another example, U.S. Patent Application Publication No. 2006/0149140by Eldridge provides a diagnostic and treatment system for patientdiagnosis and crisis management. The described system accepts a varietyof inputs, including video, sound, speech recognition, and sensorsignals. However, the system application is targeted towards a medicalcrisis type environment, such as an emergency room, where it willintegrate inputs from various devices and output diagnosis and treatmentinformation. While some consulting doctors may be remotely located, somehealth care professionals are present to operate the system and treatthe patient. Thus, again, the Eldridge '140 system does not propose amonitoring system for physiologic conditions applicable to the homeenvironment. Specifically, it can be seen that neither Wilk '278 norEldridge '140 anticipate an unobtrusive privacy-maintaining systemcapable of ongoing, day after day, monitoring of multiple individuals.Additionally, neither system provides image normalization to reduce thevariability associated with capturing images of different individuals,under a variety of lighting conditions, taking into account seasonalchanges, and other factors that would be common to capture in a homeenvironment.

The general need for physiological monitoring of individuals outside thetypical clinical environment is known. For example, U.S. Pat. No.6,205,716 by Peltz describes a modular portable video-conferencingenclosure or kiosk for facilitating remote telemedicine. However, theapparatus of Peltz '716 is intended to be equipped with sophisticatedequipment to perform ECGs and EEG, and other tests, thus enablingtelecardiology, telesurgery, and other kinds of direct medical care. ThePeltz '716 system can be as expansive as a flatbed truck and is clearlynot intended for common-day residential use.

As another non-clinical application, the prior art includes patents suchas U.S. Pat. No. 6,927,694 by Smith et al., which describe camera-basedsystems which image facial features to enable assessment of thepotential fatigue of a driver of a vehicle. Such systems can assessdriver drowsiness relative to various physiological parameters,including eye blink, head movement, facial expression, yawning, whileoperating under a range of illumination conditions. However, thesedriver fatigue assessment systems are not used to assess the well-beingor health of one or more individuals in a residential environment. Thus,these systems do not anticipate the issues (including managing privacy,unobtrusive image capture, image normalization), the opportunities, orthe design of a residential family well-being monitoring system.

Other patents, such as U.S. Pat. No. 6,611,206 by Eshelman et al., andU.S. Pat. No. 6,968,294 by Gutta et al., anticipate the need for homehealth monitoring of individuals, such as the elderly, who wouldnormally need a caretaker to protect their health. The monitoringsystems of these patents includes a pervasive array of sensors,including cameras, to enable monitoring of the subject relative tobehavior, emotional state, activity, safety, environment, and security.These systems also include devices to provide local or remote alertsconcerning the subject and his or her environment. The systems ofEshelman '206 and Gutta '294 are neither unobtrusive nor intended forgeneralized family health care. Additionally, these systems really donot provide imaging-based health assessments for multiple individualsthat address the variability that would be expected, includingvariations in age, ethnicity, ambient lighting conditions, seasonallyinduced changes in appearance, privacy, health history, and otherfactors.

Another patent, U.S. Pat. No. 6,539,281 by Wan et al., provides for amedicine cabinet or similar device that assists users in selecting,taking, and tracking their use of medications. In this instance, themedications are provided with radio frequency identification tags, andthe medicine cabinet is equipped with a radio frequency tag reader. Atouch screen flat panel display can be provided with the cabinet, as aninterface to the users. The cabinet may include a camera and facerecognition software, to enable user identification. While theintelligent medicine cabinet of Wan '281 is useful, it does not use acamera for assessing the physiological state or conditions of the users,and as such, it does not anticipate either the issues or opportunitiesthat arise from such considerations.

There are other health care devices that are more focused on the homemonitoring of health or medical parameters, rather than general behaviorand activity. As an example, international patent publicationWO2001/071636 by O'Young describes a personalized health profilingsystem intended to collect quantitative health data on individuals intheir home environments, so as to look for warning signs of potentialdisease or a changes in one's health or physical state. The datacollection is intended to be sufficiently unobtrusive that it can beundertaken during normal daily activities, such as working, sleeping, orexercising. In particular, O'Young '636 anticipates that one or moresensors are to be worn by an individual proximate to their body, tomonitor heart rate, blood oxygenation, gait rhythm, or body temperature.Similarly, international patent publication WO2005/006969 by Montvay etal. anticipates a health monitoring system that enables healthrelated-coaching of an individual who may be in their own home. Thissystem can have sensors that are worn by an individual, or implanted intheir body. Such sensors can monitor the electrocardiogram (EGG) or arespiration rate of the individual. Other sensors can be provided, forexample mounted to a wall, to monitor environmental data, like airtemperature, humidity, and other parameters. While the devices andsystems of O'Young '636 and Montvay '969 are targeted for home healthcare, they are not targeted for generalized family health care. Inparticular, they do not anticipate an unobtrusive system capable ofongoing, day after day, monitoring of multiple individuals.Additionally, none of these systems provides image normalization toaccount for the variability associated with multiple individuals,lighting conditions, seasonal changes, and other factors.

The system of U.S. Patent Application Publication No. 2003/0069752 byLeDain et al. has greater comparative relevance, as imaging is a keyaspect of the described home health-care system. LeDain '752 anticipatesa home health care system to facilitate medical care and monitoring foran individual by a health care clinician, where the clinician can bepresent or located remotely. To enable remote care, the individual ofinterest would possess an equipped medical kit, a teleconferencingcamera, and a gateway computer for data transfer. The medical kit can beequipped with various medical devices to measure vital signs, such as ablood glucose meter, a blood pressure measurement device, a bloodoxygenation sensor, or an ECG module. On the occasions that a clinicianis not present, the individual would use these devices, followinginstructions provided by the gateway computer. The video camera enablesreal-time teleconferencing between the individual and a clinician. Italso enables a clinician to record events from a visit to theindividual's residence. Ultimately, a medical professional can use thevideo images to assess the physical condition and the behavioralindicators of the individual in question. Provision is made tounobtrusively hide the video camera within a picture frame behind aphotograph. However, the photo is then deliberately removed when thecamera is used for video capture.

The system of U.S. Patent Application Publication No. 2005/0228245 byQuy, uses a similar home health care system to that of LeDain '752. InQuy '245, a user is provided with a health-monitoring device thatcommunicates to remote locations through a cell-phone wireless device(such as a PDA) to a remotely located caregiver or clinician. The healthmonitoring device can have one or more modules or sensors for measuringhealth attributes such as blood glucose or oxygenation levels, bloodpressure and heart rate, respiration, temperature, or exerciseperformance, of a human subject. A camera, which can be a separatemonitoring device, or integral with the wireless communication device,can be provided to collect visual image data via still or videoelectronic photography.

Although the systems of LeDain '752 and Quy '245 describe home healthcare monitoring systems that involve imaging, in these systems, thehealth care monitoring is intended for a previously identified subjector patient. In particular, patients who are being monitored for avariety of conditions can be sent home with a PC-based or network-basedtelemedicine appliance that can be used to connect them back to ahospital or doctor's office via ISDN, DSL or cable modem connections.Additionally, these systems employ a range of bio-medical sensors, whichtypically require physical contact to function, and where imaging isonly a secondary component.

SUMMARY OF THE INVENTION

Thus, the present invention is generally useful for aiding individualmonitoring and assessment of general well being and health. The presentinvention functions unobtrusively, perhaps on a daily basis, withconsideration for privacy, while enabling monitoring of one or moreindividuals. The system enables assessment of a wide range of generalphysiological conditions, which can be used within the family setting,or shared within a family's social network or with medicalprofessionals, as seems appropriate.

Monitoring is enabled by the use of semantic data and by imagenormalization and data assessment to generate robust image content andphysiological metrics. These data can be shared immediately or remotely,and conveyed by a variety of techniques.

BRIEF DESCRIPTION OF THE DRAWINGS

In the detailed description of the preferred embodiments of theinvention presented below, reference is made to the accompanyingdrawings in which:

FIG. 1 shows a perspective of a user of the present inventioninteracting with a system capable of providing the present invention;

FIG. 2 a shows an illustration depicting the primary elements of thepresent invention;

FIG. 2 b is a cross-sectional illustration of a portion of FIG. 2 a;

FIG. 3 is an illustration depicting the present invention configured asa network;

FIG. 4 is a block diagram depicting the primary operational elements ofthe present invention;

FIGS. 5 a, 5 b, 5 c, and 5 d are flow diagrams depicting aspects of thepreferred operational methodology of the present invention;

FIG. 6 is a picture of an eye, showing the sclera relative to the irisand the pupil;

FIG. 7 is an illustration of a reference image; and

FIGS. 8 a and 8 b depict alternate constructions for an integratedimaging and capture device, based on prior art designs that can beutilized by the system of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The apparatus and method of the present invention addresses the need fora system for monitoring physiological conditions, which includestechnologies to acquire physiological data over time, analyze the datato provide wellness assessments, validate the data and test theassessments, and provide assessments to the users. The system caninclude an integrated display and image capture device. The keyfunctional attributes of the system include the following:

-   -   It provides day after day ongoing access in an unobtrusive way,        targeted for the home environment.    -   It is designed with consideration for family privacy issues.    -   It is enabled by identification of individual users (user        provided, face recognition, audio recognition, for example)    -   It can be adapted to provide monitoring and assessments for new        individuals or new conditions.    -   It can use an integrated capture & display device, which can use        more than just visible light.    -   It is primarily used as an imaging system to collect facial        images, although it can be used to collect images of other body        regions, or can be used indirectly via peripherals or        accessories.    -   It normalizes collected images for different individuals and        takes into account changes in appearance such as might happen        seasonally (for example tanning).    -   It compiles assessments of physiological conditions or changes.    -   It can provide or enable assessments of multiple individuals.    -   It provides alerts, stores data, and enables data access.

The basic functionality of the hardware portion of physiologicalmonitoring system 300 is shown in FIG. 1, wherein a user 10 faces anelectronic imaging device 100. Generally speaking, electronic imagingdevice 100 provides image capture, and can provide image and datadisplay, or both image capture and image and data display. In theinstance of FIG. 1, electronic imaging device 100 is illustrated as a“display that sees” device that includes both an image display 110 andone or more cameras 120. The display 110 can be a computer display(desk-top or laptop), a television, an electronic book, or otherelectronic display device. As a particular example, electronic imagingdevice 100 can include a computer equipped with a video camera, whichcan be a web-camera. Web-cams are commercially available from numerouscompanies, including Creative Laboratories (Singapore) and Logitech(Switzerland). As shown in FIG. 1, electronic imaging device 100provides an output of display light 230 from display 110 in thedirection of user 10. Ambient light 200, which can be natural lightingor room lighting also illuminates user 10. A portion of this lightbecomes capture light 220, which is collected by camera 120 and focusedby a lens (not shown) onto an internal sensor array (not shown). If theambient light 200 is insufficient or sub-standard electronic imagingdevice 100 can supply illumination light 210 from an illumination lightsource 215 to illuminate user 10.

In a broader context, the hardware for physiological monitoring system300 can be understood via FIG. 2 a. The primary elements ofphysiological monitoring system 300 are the electronic imaging device100, which includes at least one camera 120, and possibly a display 110.Electronic imaging device 100 is interconnected to image processingelectronics 320, a system controller 330, a computer 340, memory or datastorage 345, a communications controller 355, and a network 360. Theimage processing electronics 320 potentially serve multiple purposes,including improving the quality of image capture of the camera 120associated with a local electronic imaging device 100, improving thequality of images displayed at a local display 110, and processing thecaptured images to aid the derivation of metrics relative tophysiological conditions. Computer 340 coordinates control of the imageprocessing electronics 320 and system controller 330. Computer 340 alsomanipulates and accesses data from memory 345, display 110, imageprocessing electronics 320, and network 360. Both image processingelectronics 320 and computer 340 can access various databases (whichwill be discussed subsequently), many of which are stored in memory 345.System controller provides various control and driver functions for alocal electronic imaging device 100, including display driver and imagecapture control functions. A variety of detectors can be provided,including an ambient light detector 140, a motion detector 142, andvarious secondary detectors 144 that can be used for measuring ambientlight or other physiological or environmental parameters. Thesedetectors are interconnected with computer 340 or controller 330.Communication controller 355 acts as interface to a communicationchannel, such as a wireless or wired network 360, for transferring imageand other data from one site to the other.

As noted previously, the principal anticipated application ofphysiological monitoring system 300 is in the residential market. Yetunfulfilled needs can be identified from a purely medical perspectiveand from a broader context of human well-being and health. Thenewspaper, USA Today, reports that the United States could have ashortage of 85,000 to 200,000 doctors in 2020, fueled not only bymalpractice insurance and other non-medical business issues impactingthe numbers of students who go into medicine, but also by 79 millionbaby boomers reaching retirement age and needing more medical care.Further, decreasing contributions towards health care from employers andgovernmental entities will mean that consumers will pay much more forhealth care. These pressures will likely force increasing health careexpenses upon consumers, which might be somewhat ameliorated ifconsumers can better assess if and when intervention by health careprofessionals is warranted.

Undeniably, the doctor shortages, the increased needs of baby boomers,and the diminishing contributions made by innumerable employers meansthat consumers increasingly have to take greater control over their ownand their family's health care. Significant care for acute conditionswill likely be shouldered by the “sandwich generation”, i.e., almost 3in 10 of those aged 45 to 64 with children in the home, who are alsocaring for a senior, according to a study based on the 2002 GeneralSocial Survey. And many of those parents who do not have their ownelderly parents living in their homes are still anxious about theirelderly parent's health, especially when the elderly live in distantlocales.

The US health care system is primarily one of “break-fix”. A consumergets a condition or a disease, and then the health care system treatsit. In fact, the majority of the health care budget is spent oncepatients have become very ill. Relatively little health care money isallocated to very early detection of deteriorating or changingconditions.

In that regard then, consumers, whether at an individual or familylevel, would be advantaged by a system that enables ongoing physiologicmonitoring. Consider the relevant example that “Mom” would benefit bytools that enable her to track the well-being and health of her kids andparents at home (or at her parents' distant home). Such a system wouldincrease her chances of noticing disease early, or other conditions(such as excessive cold or heat) that could lead to illness or disease.Although “Mom” will not likely have devices at her disposal that requiresubstantial medical training to use and to interpret (such as X-Ray orheart monitoring devices), any tools that help her discover the symptomsor evidence of a significant physiological change that has occurredwould have considerable value to her.

In particular, a range of physiological conditions or changes related togeneral well-being or health, such as nutrition, posture, weight, lackof sleep, and emotional state, might be assessed from a visual (imagebased) record. Likewise, a range of medical conditions, or the symptomsthereof, might be identified (including neurological conditions orcirculatory problems) from a visual record. Additionally, physiologicalchanges for an individual relative to a range of medical conditions(such as diabetes, heart conditions, or chronic wounds) might bemonitored or documented with a visual record. Significantly, thephysiological monitoring system 300 is intended to enable theacquisition of a longitudinal record of image and image-derived data. Asa result, the system 300 can document, and perhaps identify, changes inphysiological conditions that might evolve slowly and occur with littleawareness. Thus, “Mom” could apply her parental instincts, supplementedby both past and present data, to determine if enough change has takenplace to warrant intervention by medical professionals. These data canenable “Mom” to reach health related assessments through variousintermediate steps, such as consulting with Internet databases, hersocial network of family and friends, and other reference points, thatmay reduce the need to consult with medical professionals.

The study of the visual properties of skin is well known in the computergraphics field where it is important to understand the effects of agingand environmental factors on the way a person looks to be able toproduce an accurate simulated image of a person. This can be importantin modifying an image to simulate the effects of aging on a person as anexample. An exemplary article that shows a variety of effects to the waya person looks under different lighting conditions for the applicationof computer graphics is contained in the article by N. Tsumura et al,“Image-based skin color and texture analysis/synthesis by extractinghemoglobin and melanin information in the skin”, Proc. ACM Transactionson Graphics, SIGGRAPH 2003. While the visual properties of skin havebeen studied, the automated monitoring of a person's visual attributesover time for the purpose of tracking changes in health has not beensuggested.

The aforementioned longitudinal record of physiological data is apotential input for medical assessment that is seldom available today.As people go about living their daily lives, a given individual orfamily member, can be unaware of a physiological change until a crisisoccurs. For example, parents can miss a daughter's developing anorexiauntil it is evidenced by extreme weight loss and skin color changes.However, the accumulation of an ongoing longitudinal record,supplemented by an assessment technique, will document and pro-activelyhelp to identify, evolving physiological changes. Although system 300can obtain data on a daily basis, many wellness parameters 410 willgenerally change very slowly, and thus some wellness data can bemeasured and retained on a less frequent basis. For example, as physicalattributes such as weight or posture tend to change slowly, theassociated wellness parameters can be sought or retained on a weekly,monthly, or quarterly basis, depending on the attribute or trait inquestion and the variability associated with its measurement.

Thus, the system 300 is intended to enable the collection of a record ofphysiological data for one or more individuals. To enable this, thesystem 300 is provided with a dual-purpose device, and in particular anelectronic imaging device 100 that unobtrusively captures images of auser or subject via one or more cameras 120. Electronic imaging device100 can be a computer monitor, television, cell phone, mirror, or otherdisplay that sees the subject (with a camera 120) while the subject(user 10) is looking into the device. As shown in FIG. 1, electronicimaging device 100 is a computer, such as desktop or laptop system. Thecamera 120 can be mounted at the display edge (as shown), or beintegrated into electronic imaging device 100, such that it looksthrough the display 110 at a user 10. Whereas, the electronic imagingdevice 100 shown in FIGS. 2 a and 2 b includes a mirror 136 integratedwith a camera 120 and (optionally) a display 110. A camera 120 typicallyincludes an imaging lens 122 that provides an image onto an image sensorarray 124, through a spectral filter 126. In this case, camera 120 canlook through an aperture A, for example provided by a semi-transparentmirror 134. To aid in hiding the camera 120 and aperture A,semi-transparent mirror 134 can have a gradient reflectance, with thelowest reflectance in the center of aperture A. The semi-transparentmirror 134 can also be a flickering device that is driven electronicallyto switch between reflecting and transmitting states. Alternatelyaperture A can be an optical pinhole (<0.5 mm diameter), making camera120 a pinhole camera. In any case, cameras 120 are preferably hiddenwithin device 100, and not generally visible to the users. As shown inFIG. 3, physiological monitoring system 300 can be networked, andutilize several electronic imaging devices 100 within a residence,including both the computer monitor and mirror types. In principal, theintention is that the physiological images are unobtrusively collectedwhile the subject or subjects look into the mirror or display, whichthey are already doing to view themselves, or to view information,communications, or entertainment. These captured images can be acquiredday after day, month after month, and year after year, resulting in arich image-based representation of the subjects over long periods oftime.

Although the configuration of physiological monitoring system 300 as adistributed network is particularly advantageous relative to capturingphysiological image based data for multiple family members, variousissues regarding individual and family privacy are accentuated. Inparticular, placement of electronic imaging devices 100 as one or morebathroom mirrors is advantageous relative to the image capturing. Forexample, in a household, a mirror type electronic imaging device 100 canbe provided in the master bathroom, while another can be provided in achildren's bathroom. Considering human behavioral patterns involvingpersonal grooming, the best opportunity for capturing image data on aday after day basis could be from the mirror type electronic imagingdevices 100. Also, the most repeatable, and perhaps the best, set ofillumination conditions might be found in the bathroom setting. However,as can then be anticipated, management of user privacy, particularly inthe bathroom setting, is very important. On the other hand, electronicimaging devices 100 that are integrated into a computer, television, orentertainment station would be expected to see regular usage on a dailybasis, or nearly so, depending on the household. Although the privacyconcerns related to image capture from these non-bathroom locateddevices might be reduced, the image capture conditions may be bothinferior and more variable. In any case, various hardware and softwaredesign features can be integrated into physiological monitoring system300 to address privacy concerns and any associated variability inherentin the capture conditions.

Notably, it is not sufficient to simply capture an image, but imageassessment, enabled by image normalization, is key. Again, considering ahome environment, the appearance of family members can varysignificantly relative to gender, age, skin color, hair color, height,weight, and other factors. Likewise, the basic appearance of anyindividual can vary by season (such as tanned or sun-burnt), by behavior(including use of cosmetics, exercise, or alcohol and drug use orabuse), and by other factors. The ambient lighting can also changedramatically from one image capture opportunity to the next. In asimilar fashion, the position of an individual relative to the imagecapture device can lead to variation in the size, orientation, orplacement of the individual in the captured image. Therefore, tocompensate for these wide ranges of variables that can affect imagecapture and interpretation with unobtrusive image capture, the processof physiological monitoring employs an image normalization process 500to decrease the impact of the capture variables. In particular, thecapture step is followed by the image normalization process 500, whichmodifies the captured imagery before size or color-based wellnessparameters are derived from the image data. Processes for assessingphysiological conditions of the subjects then follow the datanormalization process. Likewise, these processes for assessing orinferring a subject's well-being must account for subject variabilityrelative to appearance, behavior, privacy, and other factors.

As then can be seen, FIG. 4, which is a block diagram, and FIGS. 5 a-dwhich are flow diagrams, together illustrate in greater detail theoperational considerations and logic of the physiological monitoringsystem 300 of the present invention. FIG. 4 particularly illustratesmany of the data processing functions that are realized through aninteraction of the computer 340, image processing 320, and memory 345,while FIG. 5 a generally illustrates the overall operational processesthat the system steps through when in use. As an example, physiologicalmonitoring system 300 can be operating according to an internal clock(not shown), such that it is shut off at night and then operates in alow energy consuming watchful state during the day.

Camera 120, ambient light detector 140, motion detector 142, and usertracking process 515 together include an image capture system 310, whichare used in a coordinated for image capture of subjects 10. As a motiondetector 142 senses that a potential subject (step 512) has entered theoperational range of an electronic imaging device 100 located in aparticular place within a residence, an initial image capture process540 is engaged. Motion detector 142 can include a sound sensor(microphone), a light intensity sensor (including a near-IR sensor), oran optical sensor that detects motion, or a combination thereof. Camera120 can also support the motion detection function, for example usingimage area histograms to detect presence and position. A user trackingprocess 515, which can employ a motion detector 142 and cameras 120,then tracks the location of the potential subject relative to theelectronic imaging device 100. When physiological monitoring system 300determines that a potential subject has entered the field of view of acamera 120, an initial image capture process 540 would cause camera 120to acquire an initial image, with assistance from the user trackingprocess 515. Then using a subject identification process 510, which canaccess semantic identity data and can employ face recognition software,audio recognition software, or other techniques, to determine whether anindividual is a known subject of interest. A good article describingface recognition techniques for video imaging is contained in thearticle by G. Aggarwal, A. Chowdhury, R. Chellappa, “A SystemIdentification Approach for Video-Based Face Recognition”, Proc. of theInternational Conference on Pattern Recognition, 23-26 Aug. 2004,Cambridge, UK. If not, by default, system 300 would stop active imagecapture without storing any image data or starting the well-being imagecapture process 550. On the other hand, if an individual is identifiedas a known subject of interest, then the system 300 would typicallyproceed with the next steps in the well-being image capture process 550for that capture event for that particular individual.

The well-being image capture process 550 is primarily a structuredprocess for acquiring high quality images within target captureconditions for lighting, subject pose, image focus, and otherparameters. Data from various databases, such as an image captureconditions database 450, a user privacy settings database 440, asemantics information database 430, and a wellness parameters database420 is used to define the target image capture conditions for a givensubject or user 10. These various types of system data, which will besubsequently discussed in greater detail, are summarized in Table 1.During the well-being image capture process 550, the physiologicalmonitoring system 300 uses data from the privacy settings database 440that associates subject or user identification with the desired privacylevels for that particular individual. These privacy settings can bedifferent for various users of the system (family members). Likewise,the physiological monitoring system 300 uses data from the wellnessparameters database 420 that identifies and quantifies any particularphysiological conditions that are tracked for a particular individual oruser 10. In a similar fashion, the semantics information database 430can provide data concerning seasonal, cultural, behavioral, and wellnessfactors that can affect image capture or wellness analysis andinterpretation. More generally, it should be understood that semanticsis defined as the study of information related to human meaning orexperience (see Table 1). Semantic information (such as events,activities, people, conditions, locations, objects, music genres) can beassociated with an informational asset (such as an image, a voicerecord, or a data file). Finally, the physiological monitoring system300 uses data from the capture parameters database 450 that isindicative of the preferred capture conditions for given individuals. Toaid in efficient system image capture, a composite set of preferredcapture conditions and images (reference images 365) for each individualcan be pre-assembled from the database data, so that each database doesnot have to be accessed on the fly during each capture event.

TABLE 1 Primary types of system data The privacy settings database 440provides privacy settings that associate subject or user identificationwith the desired privacy levels for that individual. Exemplary privacysettings provide: Support for identification of known subjects Supportfor limiting impact on non-subjects Access controls for lead usersDefining and associating privacy settings with individual subjectsDefining target images for various subjects Defining image datamanagement for privacy sensitive body regions for various subjectsDefining use of image capture alerts Defining how assessment alerts areprovided Defining how physiological data and assessments are output,stored, and, shared The semantics information database 430 providessemantics data, which is generally qualitative data concerning seasonal,cultural, behavioral and wellness factors that can affect image captureor wellness analysis and interpretation. Exemplary semantics dataincludes: Subject identity Familial relationships Age, gender, ethnicityActivities/calendar - time of day, trips, vacations Seasonal issues -such as weather, potential impact of tanning, becoming sun burnt, windburn) Personal behavioral factors - such as use of cosmetics or alcoholor drugs, exercise Dietary habits, sleep habits & quality Type of work,work habits, stress levels Use of medications and vitamins Referenceimage metrics (to support subject identification) Special physicalcharacteristics - for example, the presence of tattoos, war wounds,accident or sports injuries, or birth defects Knowledge of currentmedical state - for example, sick, depressed, broken arm, has severearthritis Knowledge of personal, familial, genetic, or historicalconditions - such as relatives with diabetes or cancer Knowledge of useof medications or vitamins The wellness parameters database 420 provideswellness parameters 410, which are principally quantitative metrics forphysiological traits or conditions, which quantify various wellness,health, and medical conditions, or physical attributes. Wellnessparameters can be based upon single or multi-point measurements, ortemporal measurements tracking longitudinal changes. Exemplary wellnessparameters 410 include: Height, weight, body type, body-mass index Eyecolor, hair color Skin color (by location, averaged) Skin texture,structure, and moisture Skin patterning Posture, gait Geometry forfacial features Eye - color (whiteness) of the sclera Eye and bodymovements (neurological) Tiredness Nutrition Emotional state Personalcare or hygiene Dental care Specific to known medical conditions, suchas Alzheimer's, anorexia, diabetes, acne, wounds, rashes Specific toknown personal, familial, genetic, or historical conditions Specific toknown behaviors, such as use of cosmetics Specific to use of medicationsReference image metrics (for baseline physiological data) Derivedvalues; such as averages, slopes, trend-line or abrupt, longitudinalchanges, frequencies, combined, confidence values The capture parametersdatabase 450 provides capture parameters 415 that are quantitativemetrics that are indicative of the preferred capture conditions forgiven individuals. Types of capture parameters include: Reference imagemetrics (including capture criteria for image acquisition) Image sizeSubject pose or orientation, subject distance Camera settings (shutterspeed, aperture, zoom position, focus distance) Lighting conditions -intensity (irradiance) and spectrum (measured data or model) Imagequality - focus and resolution Image quality - contrast & dynamic range(noise) Image quality - still images - lack of motion blur Geometry offacial features Supporting data: lens focal length, lens aberration dataComposite data: using privacy, semantics, wellness, and system data thatimpacts capture The image normalization process 500 derives and appliesnormalization or correction factors for image attributes such as colorand size. Types of normalization data include: Reference images andreference feature data or metrics (for baseline correction factors)Normalized image data Normalization confidence values Normalizationtransforms (including for size and color correction) White point Colorbalance Other correction factors (including for audio traits or bodymovement) Other system data: Reference images Identification ofreference features

Various system components, such as ambient light detector 140, motiondetector 142, camera(s) 120, and illumination light source 215 can beused to varying extents as an aid to the well being image captureprocess 550. For example, the physiological monitoring system 300 cancollect data from ambient light monitor 315 about both the lightintensity and light spectra that can be used to enhance image captureand processing. Similarly, the user tracking process 515, which cancombine face recognition software, gesture tracking, and motion trackingalgorithms, can be used to watch for subject poses that are particularlyconducive to a quality well-being image capture.

Once the well-being image capture process 550 concludes a capture eventwith the capture of one or more images that satisfy the targetconditions, the system operation progresses (see FIG. 5 a) into an imagenormalization process or structure 500. This process step principallycorrects the newly captured images for color errors and sizing errors,so that any changes that are subsequently observed in the newly capturedimages can be correctly attributed to physiological changes, as thevariable factors of image capture are compensated for. In particular,the image normalization process 500 is tasked to overcome a number ofproblems that an unobtrusive system will encounter (changes in roomlight, distance and direction of the user from the capture device). Asshown in FIG. 5 a, the overall operational process for physiologicalmonitoring system 300 then progresses to a step of calculating andupdating wellness parameters 410 and capture parameters 415. Thewellness parameters 410 are principally physiological metrics, which canbe input by users 10, or derived from the image data captured by cameras120, or calculated during subsequent analyses. Generally the wellnessparameters 410 quantify various wellness, health, and medical conditionsor attributes. Once these wellness parameters 410 are calculated andstored to a memory 345, an inference engine 400 (which is functionallysupported by a computer 340) is utilized in the subsequent step toassess the status of physiological conditions. The inference engine 400,which can be algorithm based, or utilize artificial intelligence (AI) orlearning methods, is tasked to follow and assess previously identifiedphysiological trends for its subjects (users 10). It is also intended tolook for changes, in the physiological data (wellness parameters 410),subtle or otherwise, that might be indicative of previously unidentifiedphysiological changes. Inference engine 400 also looks to reduce errors(reduce false positives) by using a health database 460 and thesemantics information database 420 to identify potential causes ofapparent changes. As an example, the inference engine 400 can anticipatepotential external changes that impact measured parameters that are notindicative of a real concern (e.g. skin tone—exertion level, sun andwind effects, or makeup). Should inference engine 400 conclude that anew issue or concern has arisen, there are several actions it caninitiate, including providing alert signals 350 to users 10.

Approaches to meeting this challenge unobtrusively involve several keyaspects. Notably, it can be expected that when a new group of users 10first start to use a physiological monitoring system 300, that theywould input various data, including user identity data, privacypreference data, wellness or health data, and semantic data into thesystem. The input can be via keyboard, digital cameras, photo scanners,voice, or other suitable methods. This initial input data would beappropriately assembled into the semantics database 430, the userprivacy settings database 440, and as wellness parameters 410 in thewellness parameters database 420.

Additionally, physiological monitoring system 300 can utilize anauto-setup process 520, shown in FIG. 5 d, to characterize controlparameters for the individual users 10 under the various conditions thatthe system must work under (such as for different times of day, userpositions in room, or room lights on or off). This process involvesmonitoring the users 10 (step 522 b), as well as the user environments(step 522 c, such as for light levels) for a suitable extended period oftime (over multiple capture events) and analyzing the resulting data sothat standard poses and lighting conditions can be determined for eachparticular individual. Step 522 b, capturing subject image data,includes sensing an individual (step 512), initial image capture (540),subject-tracking 515, which for brevity, are not shown in FIG. 5 d. Thecriteria for these capture conditions are likely determined inaccordance with pre-defined guidelines (including for lighting or pose),which can account for likely variation in the color or tone of skin 40,height, weight, user pose, and other parameters that are seen in humanpopulations. Thus, the initial input data (step 552 a) affects thisprocess. The resulting target capture condition criteria for eachindividual likely indicate a range of acceptability for each condition,rather than a unitary condition of acceptability. These target criteriaquantify factors including image size, user pose, and lighting. Thecapture condition criteria are then expressed as capture parameters 415in a capture parameters database 450. In effect, the auto-setup process520 defines the preferred image capture criteria for each particularindividual, which becomes the standard or baseline that the well beingimage capture process 550 uses in seeking to acquire quality images. Thepredefined capture criteria (step 522 a) for an individual aredetermined by associated privacy settings, semantic data, well-beingparameters, and the capture parameters, either individually or incombination. The auto-setup process 520 can accept input data (indicatedby dashed lines) from external sources (steps 522 a and 522 e),including third party entities, which can for example, define newphysiological metrics (wellness parameters 410) to be monitored.

The subject data and environmental data acquired during the auto-setupprocess is analyzed (step 522 d) to derive target values to be usedduring subsequent system operation. During this auto-setup process 520,the physiological monitoring system 300 can also create one or morereference images 365 for each subject (step 522 e). For example, mostcommonly, a head and shoulders image of an individuals face 25 seendirect on, will be the primary reference image (see FIG. 7). The mouth60, hair 65, eyes 30, and other facial features should be clearlyvisible. Other reference images, such as direct-on views of the head andtorso can be generated. The baseline reference images, and the baselinedata derived from the reference images, can be established by variouscriteria (based on pose and lighting, for example) and can be obtainedfrom (selectively) averaged data acquired during the duration of theauto-setup process 520. Although the reference images themselves can bestored in memory 345 as system data in step 522 f (or as captureparameter data 415), reference image parameters (including physiologicalmetrics) can also be derived and stored as accessible capture parameters415 or wellness parameters 410 in the appropriate databases to aid theimage capture process. The reference image parameters generally quantifyphysical attributes or reference features 367 of the subjects. Mostreference features relate to physiological attributes that can beexpected to be nominally stable or static for relatively long periods oftime (such as months or years). For example, the distance between theeyes represents a reference feature 367, that when quantified, is stablein adults, but which can also be used to scale an image or to monitorthe growth of a child. The reference images 365 can be broadly used bythe system 300, during subject identification, well-being image capture,image normalization, wellness assessment and inference, and wellnessreporting. Facial reference images, or metrics related to referencefeature 367 derived there from, can also be used as semantic data tosupport subject identification (via process 510).

Normalization correction factors, which can be derived from thereference images 365, and which can relate to capture parameters 415also derived from the reference images 365, are also useful to thesystem 300. As an example, for the reference feature 367 of eye-to-eyedistance, the associated wellness parameter 410 would contain the knownphysical distance, the associated capture parameters 420 would contain arange of acceptable sizes (related to subject distance) and head posesthat would be suitable for a capture event, and the associatednormalization correction factors would contain scaling factors, based ondistance and pose to adjust newly captured images to the same scale asthe reference images 365. Nominally the normalization scaling factor ortransform in the case of size would be a multiplying factor to adjustthe reference feature size in the image captured by the camera to equalthe real physical size. In addition to obtaining baseline normalizationtransforms, the auto set-up process 520 can be used to acquire baselinenormalization confidence values, which are statistical metrics of thequality of the normalization. The confidence values (or confidencelevels) are a measure of confidence assigned to the value of anattribute (in this case, a normalization transform), which are oftenexpressed as a percentage (0-100%) or a probability (0-1). Thesetransforms can for example be implemented by a variety of imageprocessing operations that can include but are not limited to pointbased image processing techniques such as shifts, multiplies, matrixoperations, polynomial operations and single or multi-dimensional lookuptables. Of course a variety of normalization transforms may beapplicable to one or more types of input data. For example, if thesounds captured are to be normalized to remove environmental factorsthat were present at the time of capture (such as a baby crying or atrain passing) frequency based normalization may be useful. In the casethat multi-spectral or non-visible data image or point data has beencaptured similar normalization transforms and confidence values will becalculated.

Baseline values or metrics for various other reference features 367,beyond just the eye-to-eye distance, can be established during the autoset-up process 520. For example, a baseline metric for the distance fromthe eye-to-eye centerline to a line crossing the upper edges of thenostrils (nares) can be measured to provide a vertical size scaling andstored as a wellness parameter 410. Baseline values for wellnessparameters 410 can be established for other physiological attributes,such as skin color, color (whiteness) of the sclera 32, weight, posture,body-mass index, using imaging or other sensing means, as appropriate.

It is noted that baseline metrics or wellness parameter values 410 canalso include temporal data that characterizes a reference feature 367.In particular, the temporally measured reference feature data relates tophysiological attributes that can be expressed within the typicalcapture events, which are likely seconds or minutes in duration. Forexample, captured video data can be used to acquire temporal measuredwellness parameters 410 for physiological attributes such as such as eyemovements (blinks/minute, side to side motions/min.), hand tremors (mmmovement/sec), gait, or other attributes that can be indicative ofneurological or motor control conditions. Likewise, microphones (144)can be used to collect baseline audio data, and particularly voice datafor each subject 10. The non-linguistic cues that a speaker uses toguide listeners and signal intent are collectively called prosody.Prosody includes such factors as voice pitch, pacing, and loudness andcan occur consciously or unconsciously. As an example, speech analysistargeting voiced speech, which has a strong harmonic spectral structure(basically the vowels), can be a fast and efficient approach. Baselinestatistical values for various temporal voice reference features 367,including frequencies, pitch, voicing rate, segment durations can bemeasured and then later used as wellness parameters 410 or asidentifiers for subjects 10. As one approach, an auto set-up process 520targeting voice data can include having subjects 10 read or recite vocalcontent that utilizes significant regions of their vocal range. Likewisean auto set-up process related to gait can include acquiring videoimagery of the subjects 10 walking with a normal stride. These methodscan be repeated during multiple auto set-up capture events. In addition,data from natural, rather than deliberate behavior or activities duringauto set-up capture events can be acquired and used.

In summary, the auto set-up process 520 depicted in FIG. 5 d nominallyestablishes a collection of baseline data for a system 300, as appliedto one or more users 10, under a range of variable capture conditions.This baseline data includes baseline reference images 365 and referencefeatures 367, baseline capture condition data (capture parameters 415),baseline wellness parameters 420, baseline normalization transforms, andbaseline normalization confidence values. The effects of capturecondition variability can be reduced by having an extended auto set-upprocess 520, during which sufficient data is captured under variousconditions, that data variability is statistically reduced, for exampleby averaging. For example, relative to color normalization the whites ofthe eye (sclera 32) can be particularly useful. The auto set-up process520 can establish baseline values for scleral color or whiteness,normalization transforms, and normalization correction factors. Thesevalues can be averaged, to reduce the potential impact of the eyes beingbloodshot or the sclera may be segmented to eliminate contributions ofthe vascular system to the color of the sclera. Additionally, the users10 can also be provided with guidelines or instructions regarding theauto set-up process 520, to direct them through a range of captureconditions. For example, these guidelines can direct users 10 through arange of poses and lighting conditions on multiple days, to aid thesystem in assembling robust baseline data.

To better understand the operation of physiological monitoring system300, FIG. 5 b illustrates the well-being image capture process 550 andthe image normalization process 500 in greater detail. Image captureusing reference image data (previously acquired during an auto set-upprocess 520) can then proceed as follows. For example, each day, thephysiological monitoring system 300 can obtain images of each subject10. When the physiological monitoring system 300 identifies a targetsubject 10 within its field of view, the well-being image captureprocess 500 is engaged. During this process, the system willcontinuously monitor the subject 10 to obtain at least one image of thesubject 10 under the preferred target conditions for that particularindividual. These preferred conditions are quantified by the referenceimages 365 and the capture parameters 415 stored in the captureparameters database 450. Camera 120 can be capturing (step 552 a) one ormore still images, or video images, while the subject 10 is monitoredwith the user tracking process 515. Camera 120 would adjust focus of theimaging lens 122 to ensure that the captured images are sharp. A streamof video images can be directed to temporary memory, and key-frame, keyvideo extraction or video summarization can be used to identify andretain one or more images that meet the target criteria. Audio data,including voice data, for the subject 10 can also be recorded, as theopportunities arise.

The well-being image capture process 550 seeks to acquire images (step552 a) that correspond to one or more reference images 365 for a givenparticular individual, and satisfy the capture criteria (step 552 b),expressed as the capture parameters 415, relative to variables such asimage size, subject pose, motion blur, and lighting. As one example,during the well-being image capture process 550 the system (step 552 b)would look to acquire one or more images of an individual in a pose thatis most similar to a standard pose(s) of a reference image 365. If thepose obtained is within predetermined tolerances, and the other capturecriteria are met, image data will be captured for that capture event andthe system can proceed to the normalization process 400. If the poseobtained is outside the target range, the system can store the bestimage it obtains, and wait for an opportunity to collect a better image(that day) for a given particular individual. Ultimately, sub-par imagedata with an out-of-target (within some range) lighting or pose canstill be used for at least some purposes. For example, while thelighting available for a given image can be inadequate for acquiringgood color images, thus preventing useful color based physiologicalassessments, other parameters, which do not rely upon color imaging,such as posture, gait, or eye movements, can still be assessed.Additionally, face geometry algorithms can salvage some images that aresub-par relative to pose. Such algorithms can be used to estimate theangular displacement of a face 25, and then using 3D reconstructions ofthe face 25, a 2D representation can be calculated that enablesappropriate measures. In particular, the face geometry algorithm can usedistance reference features 367, such as the calculated distance betweenthe eyes 30 to estimate the facial orientation. This derived value canbe compared to the eye-to eye distance derived from the reference images365 for that subject. Other capture data, such as the focal length ofthe imaging lens 122 and the calculated distance to the subject can alsobe used to assess or correct for image pose or orientation, as well ashelp verify the eye-to-eye distance. Additionally, these 3D geometrymeasurements and corrections can be based on data collected from amultiplicity of image (video) frames. If the physiological monitoringsystem 300 has multiple capture cameras 120, including possibly withstereoscopic imaging capability, then images from a combination ofcameras can be used to acquire an image within the target pose ororientation range. Stereoscopic imaging can also be used to measure thedistance to the subject either with a dual lens rangefinder module orthrough a range map generated from a disparity map produced from thepixel offset information for a set of images captured by multiplecameras with similar fields of view. A dual lens rangefinder module isdescribed in U.S. Pat. No. 4,606,630, which was issued to Haruki. Adescription of a method for producing a range map from a disparity mapis described in U.S. Patent Application Publication No. 2006/0193509published in the name Criminisi.

In large part, the target criteria for a good well-being image captureanticipate the needs of the image normalization process 400. As will bediscussed, image normalization relative to color is also very important,and depends on the derivation of color calibration data. Of course, theapparent image color is very dependent on the lighting conditions. Tobegin with, the physiological monitoring system 300 can monitor roomlighting conditions to verify that the capture is taking place understandard conditions. For example ambient light detector 140 can measurethe light level in the room to determine whether the light intensity ishigh enough that the signal to noise levels for the image data should beacceptable. Ambient light detector 140 can also include spectralfiltering or spectral dispersion devices (such as dichroic filters ordiffraction gratings) to enable measurement of the optical spectrum ofthe ambient light 200. The spectra of the capture light 220 can alsospecifically be measured. Depending on the color correction criteriaused, it can be sufficient for the physiological monitoring system 300to use the spectral data simply to estimate a blackbody colortemperature that approximates the room lighting. For example, typicaldaylight solar radiation approximates a 5900 K blackbody source.Alternately, spectral measurements can be obtained at a few choicewavelengths so that the physiological monitoring system 300 can assessthe degree to which the ambient light 200 or capture light 220 includescommon residential lighting spectra (such as from sun-light,incandescent lights, fluorescent lights, or LED lighting), eitherindividually or in combination. For example, an effective light sourcemodel can be assembled by determining that the ambient light 200 at agiven moment is 25% daylight and 75% fluorescent lighting. Finally, theambient light detector 140 can include a monochromator or aspectro-radiometer, to obtain detailed spectral measurements. A newlycaptured light source spectrum or model can also be compared to priorRGB or spectral data and color correction data or normalizationtransforms that can be maintained and updated for capture from a givenelectronic imaging device 100. If a new light source is detected in theenvironment, the auto setup process can be reinitiated to establishbaseline capture conditions that include the newly detected lightsource. This data can then be included as capture parameters 415 in thecapture parameters database 450. In general, the different spectraavailable at a given location are quite variable and can change slowly(for example, seasonally) or quickly (for example by turning on or off aroom light). A measured ambient light spectra or light source model canthen be used to derive white point and color correction factors that canbe applied to newly captured images during the image normalizationprocess. Alternately, relative color measurements can be accomplished onthe scene background to identify changes in the spectral characteristicsof the lighting for the environment over a period of time.

As can be seen, various methods for measuring or deriving the intensityand spectra of the ambient and capture light are possible. The qualityor accuracy of both the derived capture parameters 415 and the derivednormalization transforms can depend on the measurement methods. Thus,normalization confidence values can be statistically derived, which areassociated with the measurement method, the capture conditions presentat a given capture event, and the type of normalization transforms Forexample, color correction normalization transforms can have associatedcolor correction normalization confidence values, which can be differentdepending on the quality of the spectral measurement method used.

Exemplary capture parameters 415 (see Table 1) can be the intensity ofthe ambient light 200 in mW/cm² or in lumens, the focal length of theimaging lens 122 in mm, the distance from the camera to the subject inmm, and the spectra of the ambient light 200 of the capture light 220.The white point correction data can be expressed in various ways,including as CIELAB or tristimulus coordinates. It is noted that thecolor corrections and white point derived from measurements of theambient light 200 in a room may not actually represent the light spectrafalling on the subject, or a portion thereof. Physiological monitoringsystem 300 can use a tracking limited angle light collector device tomake sure that the ambient light 200 collected for measurement is indeedreflected from a target area (such as a face) of a subject 10. Thesystem can also measure the spectrum of the light collected by theimaging lens of camera 120. Physiological monitoring system 300 can alsobe provided with an illumination light source 215 that suppliesillumination light 210 with an expected spectral profile.

Taken together, the well being image capture process 550 principallyincludes capturing (step 552 a) one or more images of a subject thatsatisfy known conditions (step 552 b) for lighting and focus, targetconditions for image pose, and expected target conditions for image size(using the eye-to eye distance, for example). The newly acquired imagesfor a given particular individual from a capture event are thencertified as acceptable (step 552 c), stored in memory 345, and passedto the image normalization process 500. Updates to the captureparameters database 420 can also be made during the well being imagecapture process 550, to record the capture parameters 415 correspondingto the new image captures. The image normalization process 500 (see FIG.5 b) is structured to derive (step 502 a) and calculate normalizationtransforms or correction factors for image attributes such as color andsize, validate the derived normalization data (step 502 b), and thenapply (step 502 c) these transforms to the newly acquired images.Various normalization confidence values can also be calculated toindicate the quality of the captured data. The image normalizationprocess 500 can also include calculations of updated values for somecapture parameters 415, which will then be retained in the captureparameters database 420.

Size normalization of the captured images will be of importance to thewell-being monitoring system. For example, the derived distance betweenthe eyes 30 for a newly captured and certified image can be compared tothe expected sizing information obtained from the reference images 365collected during the auto-setup process 520. Corrective factors can bederived and applied to compensate for parallax errors related to thesubject pose during image capture. Corrective factors can also beapplied which compensate for the aberrations (such as distortion) of theimaging lens 122 of camera 120. The entirety of the new image can bescaled accordingly. Distances to various facial features, such as mouth60, nose, ears, as well as the horizon edges of the cheeks can then becalculated.

Relative to size normalization, it can be desirable to maintain multiplesize references using multiple reference features 367 related toattributes for size. For example, because of pose issues, a verticalmetric or correction factor, such as the distance from the eyecenterline to the centerline of the upper nostril edges can be used. Theassociated size normalization confidence values (used in step 502 b) canhelp the system select which normalization transforms to apply. Likewisetracking one or more reference features 367 and the associated metrics,both horizontally and vertically, for the torso and other body regionscan also useful. The use of multiple size related reference features 367is useful not only to obtain robust size normalization given posevariability issues, but also to provide additional wellness parameters410, which for example can enable the physiological monitoring system300 to monitor changes such as the weight or growth of an individualunobtrusively.

The color correction or color normalization process is comparativelymore difficult. As previously stated, color corrections based onspectral measurements of the ambient light 200 can be obtained (step 502a), validated (step 502 b), and used (step 502 c). However, there arealternate approaches for obtaining color correction data directly fromthe image data. That is, the color correction portion of the imagenormalization process 500 for the physiological monitoring system 300can key on reference features 367 of the human body 20 that are lesssusceptible to external changes from factors such as sun, wind andmakeup. In particular, the sclera 32 of the eye 30, shown in FIG. 6,which is more commonly known as “the whites of the eye”, can bereasonably expected to be white on a repeatable basis. The sclera 32appears to extend from the edge of the iris 34 to the edge of the eye30, but it actually surrounds the cornea all the way back to the opticnerve. The sclera 32, which serves as a protective outer coat for theeye, is a tough, leather-like white tissue which predominately includesType I collagen. Collagen is an elongated or rod-like molecular proteinthat forms in fibrils, which in turn are organized into larger collagenfibers. Although collagen has minimal light absorption, the size of thecollagen fibrils and fibers is such that light in the optical spectrumis broadly scattered for all wavelengths, including the visible ones. Asa result of the broad-spectrum light scattering, collagen appears whitewhen viewed externally.

To enable this, the face recognition algorithm of physiologicalmonitoring system 300 can use an algorithm that detects the eye 30, theiris 34, and pupil 36, within an image to help locate the sclera 32. Asan example, aspects of commonly assigned U.S. Pat. No. 7,058,209, MethodAnd Computer Program Product For Locating Facial Features, by Shoupu etal., can be used for this purpose. Once the sclera 32 has been locatedwithin the image, the pixel data can be extracted and color data, colorcorrection normalization transforms, and normalization confidence valuescan be calculated. The detected color difference between the white pointmeasurements for newly captured images can be compared to the whitepoint targets derived from the data from the reference images 365collected during the auto-setup process 520. Both white point and colorcorrections can then be applied to the newly acquired images so that thecorrected skin tones are true.

The sclera-based white point color differences, normalizationtransforms, and associated confidence values can also be compared to thecolor corrections that can be derived from the spectral data measuredfrom an ambient light detector 140 having a spectral measurement device.Again, both current and historical color correction data from spectralmeasurements can be used for comparisons. If sclera-based andspectral-based color calibrations are being used concurrently for newimage captures, then the derivable color corrections for a given imagecapture should match, within some tolerance range, provided that theconfidence values are statistically sufficient (validation step 502 b).But, for example, if the derived color corrections do not match, theexistence of such differences can be indicative of a physiologicalchange in the sclera of the subject, particularly if the scleral-basedcolor corrections show changes over time, while the spectrally derivedcorrections are generally constant. If comparable color changes are seenfor multiple subject or family members being monitored by a system 300,then such differences can be indicative of a system error or amacro-environmental change. As can then be anticipated, it can be usefulto maintain and track both normalization transform and confidence valuedata over time.

Although color or whiteness of the sclera 32 is a good color correctionreference point, there are other potential color reference features 367related to other anatomic attributes for color. For example, other bodyfeatures, such as the tongue or teeth, which are less prone to colorchanges from external factors than is the skin 40, can also be used forderiving color calibration data. Admittedly, teeth coloration is verydependent on the quality of dental care and can vary somewhat inresponse to foods or beverages that have been consumed prior to imagecapture. Alternately, color components of skin tone that tend to becommon to all ethnic groups, regardless of their apparent skin color,can be measured and used. Given that such features can be consideredinvariant relative to color, they can be used to calculate colornormalization parameters for imagery captured under the various standardlighting conditions (light level and color temperature). Taken together,spectral measurements, scleral “whiteness” measurements, tongue colormeasurements, or other such measurements can be used in combination toreduce the risk of errors in color calibration. Nominally, the colornormalization for a given capture event will use the available colorcorrection transform data having the highest confidence values. However,the highest values may be too low for accurate color normalization ofall the color data, but can still be sufficient for some color-relatedphysical attributes. Under such circumstances, the normalized data canbe tagged to indicate the quality of the normalization. As anotherpoint, if all the color related physical attributes being tracked for agiven subject do not require particularly accurate normalizationmethods, an easier, but sufficient method could be the regular method ofchoice.

It is noted that neutral or color reflectance standards can be providedwith the system 300, to enable validation of the color correction.However, as these reflectance standards can be lost, marred, orotherwise inconvenience the user, such an approach is not preferred, andmay be best suited for the auto set-up process 520. In any case, whitepoint and color correction terms can be derived by various methods, andthen retained as additional capture parameters 415 that are retained inthe capture parameters database 420.

Following the operational process of physiological monitoring system 300diagrammed in FIGS. 5 a and 5 b, the well-being image capture process550 results in one or more newly acquired images being certified (step552 c) relative to factors such as size, pose and lighting. Once theimage normalization process 500 has obtained color correction data, sizecorrection data, and other correction data (step 502 a), these data canbe validated (step 502 b) and applied (step 502 c) to correct ornormalize the newly acquired image of an identified subject. In manycases these changes are advantageously applied on a pixel-by-pixelbasis. Corrections for color, size, and resolution errors caused by theimaging lens 122 can also be applied as necessary. As a potentialfurther step, then newly captured and normalized images can betransformed from 2D image views to 3D image views using an appropriatealgorithm. In some cases, well-being analysis and assessments, as wellas image presentation, can be improved by working with 3D imagery.

Although the image normalization process 500 has been specificallydescribed as an image-based process that is focused on color and sizeissues, normalization has broader purposes in the present invention. Forexample, temporal data related to physiological attributes such as gaitor eye movements or hand movements can be extracted from the video imagedata, normalized for current capture conditions, and retained to supportassessments of mechanical or neurological conditions. Likewise, thesystem 300 can include microphones (a type of secondary detector 144)and an audio processor within computer 340. Audio data, and particularlyvoice data can be collected during a capture event, and the resultingdata can be normalized in intensity or frequency space to enablesubsequent derivations of wellness parameters 410 and wellnessassessments.

The image normalization process 500 is typically followed by thecalculation of wellness parameters 410, which can be undertaken by acomputer 340, either as an intermediate step prior to the operation ofthe inference engine 400, or as a first process of the inference engine400. As shown in Table 1, the semantics database 430 can contain datathat a given subject has known or suspected physiological conditions,such as Alzheimer's or anorexia. The wellness parameters database 420can then utilize both general and condition associated wellnessparameters 410 appropriate for tracking the physiological conditionswithin the capabilities of the system 300. The wellness parameters 410,such as the examples listed in Table 1, are effectively the bookkeepingmeans for quantitatively tracking physiological conditions, supported byongoing calculations of wellness parameter values with succeedingcapture events. New values for the wellness parameters 410 can bedirectly calculated from the collected and normalized image (or audio orother) data. The wellness parameters can also be derived by more complexmeans, using data from combined data sources or complex mathematics. Insome cases, given wellness parameters may not be accurately known, ormay be known to change with known ranges. This uncertainty can beexpressed with ranged values, error bars, or with wellness parameterconfidence values.

As one example of calculating wellness parameters 410, a color correctedimage can be analyzed spatially to look for both localized and macroregions of common or related colors. In particular, one standardwellness parameter 410 can be the average color of the skin 40 of theindividual's face 25, or within one or more areas of the face 25. Afurther wellness parameter 410 can be a slope of the average skin colorfor an area over a period of time, as a means of looking for trend-linetype changes. It is expected that system 300 will have a set of bothstandard and individualized wellness parameters 410 that it checks foreach subject. Other facial standard wellness parameters can include theskin color below the eyes, the location of hairline, or the facial widthacross the horizon lines of the cheeks. Standard wellness parameters 410can also be derived for other parameters such as weight and posture,relative tiredness, emotional state, personal care or hygiene, or dentalcare, depending on the image capture and algorithms being used. As anexample, personal care can be tracked with a wellness parameter thatcombined measures of hair care (length, neatness), dental care(cleanliness, tooth brushing activity), skin care (cleanliness), andclothing state (neatness, quality). Other standard wellness parameters410 can relate to tracking longitudinal changes in any of the wellnessparameters 410. Some wellness parameters 410 can be tags to identifyknown or suspected user conditions, such as tracking or identifying thata given subject 10 has diabetes or Alzheimer's. Additionally, customizedor individualized wellness parameters 410 might include trackinglocalized color changes related to acne, rashes, or wounds (acute orchronic). Other customized wellness parameters 410 can relate to knownfamilial genetic conditions or historical tendencies, as well asmonitoring for proper use of some medications. Wellness monitoring, andthe accompanying wellness parameters 410 can also be provided fortracking the effect of skin treatments and cosmetics on the color,texture, moisture levels, or sub-surface structure of skin.

Thus wellness parameters 410 can be values quantifying direct physicalattributes, or they can be derived values, such as averages, slopes,trend-line or abrupt threshold values, longitudinal change metrics,frequencies, and other combined values. Derived wellness parameters canalso include error bars or wellness parameter confidence values. Thuswellness parameters 410 will take various forms, depending on thephysiological conditions being monitored or the calculations used. Forexample, body weight data can be calculated and retained as number in lbor kg units. Skin color, for example, can be calculated and retained asnumbers in units of hue angle and color saturation. The color of thesclera 32 can be calculated and retained as a correlated colortemperature value or in terms of CIELAB color coordinate space values orvalues of other commonly used calorimetric spaces. As another example,skin texture can be tracked using measures for the surface roughness,roughness orientation, and elevation of the skin compared to thesurrounding skin. Standing posture can be tracked relative to thecurvatures of stooped or slumped shoulders and the curvature of thesmall of the back.

Accordingly the physiological monitoring system 300 is maintaining andtracking capture parameters 415, semantics data, normalization data, andwellness parameters 410 in the respective databases. This historicaldata can be retrieved and used for the calculation of updated wellnessparameters and trend-lines thereof (for capture parameters and wellnessparameters). The physiological monitoring system 300 can then update(see FIG. 5 b) the wellness parameters database 420 with the newlycalculated wellness parameter data (410), including updates to anyongoing trend-line tracking. In one context, many of the wellnessparameters 410 can be considered to be a specialized type of imagemetadata that are specifically related to health and well-being.Although many wellness parameters 410 are linked to associated referenceimages 365, these data are not likely stored as metadata in the imagefile structure, as a database structure will better facilitate the dataaccess required for the various calculation and comparison processesinternal to physiological monitoring system 300.

Once wellness parameter calculations are completed from data acquiredfor a particular individual during a capture event, individual wellnesscan then be assessed using the inference engine 400 (see FIG. 5 c),which is a specialized program or algorithmic structure (implemented insoftware or hardware) that looks for potential physiological changes. Inparticular, it can examine (step 402 a) the ensemble of wellnessparameters 410 both individually and in combinations, to look forphysiological trends or changes. It can compare (step 402 a) any newlyderived wellness parameters 410 to both the wellness parameters 410derived from the reference data, and to the longitudinal record for thewellness parameters 410, including the trend-line type wellnessparameters 410. The inference engine 400 can also compare the newlycaptured image data to the data of the reference image 365 and thelongitudinal record of images (and reference images 365) for thatindividual. In effect, the inference engine 400 is determining whetherstatistically significant changes in the wellness parameters 410 can beassociated with physiologically significant changes. The image andhealth data assessment process conducted by the inference engine 400 isintended to watch for both subtle and dramatic changes that can beindicative of physiological issues related to an individual'swell-being. The inference engine 400 can analyze (step 402 a) whethercurrent wellness parameters 410 are stable or trending in a knowndirection, for example monitoring the slope, direction, and locations ofa color change over time. The inference engine 400 can employ athreshold or step function change criteria to identify a trigger (step402 a); that is a tag that a notable physiological change may haveoccurred. While thresholds, triggers, and action limits are specifiedand acted on by the inference engine 400, they can be expressed in termsof wellness parameters 410 that the physiological monitoring system 300can calculate and monitor on an ongoing basis. Again, wellnessparameters 410 can be used for multiple uses. For example, a skindryness wellness parameter 410 can be compared to metrics for cosmeticreasons or for health reasons, such as related to the potentialformation of wounds, and thus different test action limits may apply.

The inference engine 400 can also compare (step 402 a) wellnessparameter data 410 of one individual to that of another, to look, forexample, for a trend in physiological changes that might be common toseveral family members. If the inference engine 400 does not identifyany significant changes in the wellness parameters data 410, includingthe trend line data, that is associated with a set of newly acquiredimages, then the inferring efforts can stop for that capture event.

When the inference engine 400 identifies potentially significant changesaccording to its known standards for the wellness parameters 410 ittracks, a series of validation and assessment activities ensue. As oneaspect, the ongoing changes, following either a trend line or trigger,can be assessed via a health database 460 that documents a range ofknown health and medical conditions. Such a database can be stored andmaintained locally with the physiological monitoring system 300.Additionally, an externally updated database can be accessible via anetwork (such as a mobile communications network or the Internet), whichthe physiological monitoring system 300 can access via network 360.Access can occur automatically on an as-needed basis (the system hasdetected a change but lacks clarifying data), or by a schedule, or bydirection of a user 10. The inference engine 400 can then access thesewellness or health databases 460, and look for data regarding thepotential significance of changes related to the identified trends andtriggers. The inference engine can initiate a new wellness parameter410, which can be back calculated if the data is available, that is moreapplicable to a new concern than were the pre-existing wellnessparameters 410. Likewise, as new medical or health related knowledgebecomes available, the physiological monitoring system 300 can beupdated to add new wellness parameters 410 to the list being tracked, orto apply the pre-existing data towards screening for a new concern.After collecting relevant data from the health database 460, thesemantics database 430, and the capture parameters database 450,inference engine 400 can test (step 402 b) the trend lines or triggerdata to see if they still appear significant.

Additionally, the inference engine 400 can use a process for testing forfalse positives in its assessments of physiological conditions (step 402c of FIG. 5 c). A variety of system data, including semantic data,wellness parameter data, capture condition data, normalization data; canbe utilized as part of false positive testing. As an example, falsepositive testing can access the semantic database 430 that maintainsdata regarding factors (see Table 1) such as subject age, travelschedule, exercise patterns, medication usage, or the use of cosmetics,that can effect skin color and other physiological factors orconditions. Thus, as shown in FIG. 5 c, if the inference engine 400detects an apparent physiological change, it can access the semanticdatabase 460 and look for potential causes of a false positive. Semanticunderstanding can also provide data regarding time of day and year thatcan be indicative of lighting conditions and changes. Of course, thesemantic database 460 may not be aware of events or activities that mayhave caused a potential physiological change. For example, the semanticdatabase 460 may be unawares of a vacation trip to the Caribbean, or theuse of a tanning bed, that results in a “sudden” skin color change thatis subsequently detected by the physiological monitoring system 300.Inference engine 400 can infer that a particular type of color changemay be a result of tanning and then flag the physiological monitoringsystem 300 to watch to see if the tan fades over time. Alternately, thephysiological monitoring system 300 can query a local user concerningthe significance of the observed change.

As another example, the inference engine 400 can access the captureparameters database 450 and the normalization data as part of the falsepositive testing process (step 402 c). In particular, apparent changesthat are due to variations in capture conditions rather than wellnesschanges should be identified to reduce false positives. As examples,color changes or size changes can be attributable to either differencesin capture conditions or changes in physical attributes. Thus to enablerobust testing for false positives, the system 300 preferably maintainsand tracks multiple color capture parameters 415 and color correctionfactors (such as from the sclera, skin, or measured spectrum) andmultiple size based capture parameters 410 and size correction factors(such as eye-to-eye, or eye to nose). The previously mentionednormalization confidence values can play a key role here, as theinference engine can examine both current and prior normalizationtransform values and associated normalization confidence values toascertain whether changes in the capture conditions or systemperformance may be effecting the calculated wellness parameters 410 orwellness inferences. For example, the inference engine 400 candetermine, with high confidence (high probability) that a normalizationtransform has shifted significantly from nominal. Alternately, theconfidence values associated with a given normalization transform can bedeemed low enough that any resulting inferences are suspect. Recognizeduncertainty in the wellness parameters 410 can also be factored in. Itis noted that capture condition variations, whether abrupt or trend linechanges, can be expressed in either the capture parameters 410 or thenormalization transforms. At times a user 10 will need to be alerted tochanges in the physiological monitoring system 300, ambient conditions,or other factors that are affecting the functional performance of thesystem. Inferences concerning physiological conditions can also bereported to users with commentary concerning the confidence level of theinferred conclusions.

The physiological assessment steps and the validations steps (402 b and402 c, using semantic, capture, wellness, and normalization data) can beconducted sequentially, iteratively, and in various orders. In this way,imagery can be monitored for changes in skin color or hair appearancethat might indicate a change in one or more physiological conditions.For example, bluish skin hue can be indicative of eating disorders orside effects from drugs. As another example, any trends in the colorcorrection normalization parameters that might suggest a gradual changein the color of the eyes 30 or sclera 32 can be evaluated to determineif an underlying condition that manifests itself in eye color has beendetected. Of course, these assessment and validation steps can also beapplied to non-image-based data collected by the system 300.

If the inference engine 400 concludes a capture data assessment processwith a determination that it has potentially detected a meaningful trendor change for a given particular individual, then physiologicalmonitoring system 300 can notify a local or remote user 10, a clinician,or other party. Most simply, the system can provide a local alert 350,using an audio or visual cue, for example provided through a display 110that is built into an electronic image device 100. Physiologicalmonitoring system 300 can also send an e-mail alert to one or morepre-determined e-mail addresses. Alerts and reports can be provided withvarying degrees of urgency, ranging from “please take vitamins” or“please get less sun exposure” to harder recommendations, such as“please discuss with your physician next time you see them” to “pleasesee your physician now”. As another option, a user 10 can be asked toposition themselves in front of a system camera 120, so that additionalimages can be obtained.

Additionally, if relevant, a standard medical data format, such as foran electronic patient record, can be used. For privacy reasons, thedetailed reports may only be accessible by any designated lead users,such as “Mom”. A display 100 can subsequently show image data andwellness parameters 410 illustrative of the potential concern.Physiological monitoring system 300 can also provide a compiled reportwhich can include a set of relevant images and image data, accompaniedby one or more tables of wellness parameter data, or one or more graphsof wellness parameters 410 showing trend lines or abrupt changes, andcommentary on the potential meaning or significance of the data.However, as the hard data may be difficult for many users 10 to absorb,the system can also provide a visual journal 370. Also, in some cases,the observed physiological changes can be significant, yet subtle.Therefore, this visual journal 370 can provide an image sequence thatillustrates the observed changes over a relevant period of time. Thevisual journal 370 can employ image animation or morphing to aidvisualization of the changes. The animation or morph can be derivedeither by changes in the shape of the face (such as caused by weightloss or weight gain) deforming the underlying mesh model, or changes inface color or texture or other characteristic shown as changes to theskin mapped over the underlying mesh. The animation or morph can be usedto condense changes that occurred subtly over long periods and show themvisually in a few moments. As another assist, a report or visual journal370 can depict physiological changes with unique color change imagesthat use boundary outlining, exaggerated color, or false color tohighlight the observed changes. The visual journal 370 can be suppliedas an electronic document that can be printed locally. A familycaregiver, parent or physician can use (step 402 d) the visual journal370 or other report to determine when a symptom or change manifesteditself, and how the physiologic condition changed over time or withmedical intervention.

As noted previously, personal and family privacy management is a keyaspect of the physiological monitoring system 300. Table 1 summarizes arange of privacy settings that can exist within the privacy settingsdatabase 440. The privacy issue interacts with the image captureprocess, the data management process within physiological monitoringsystem 300, and the data reporting process. To begin with, inconsidering the range of variation in people's height, particularly fromchildren to adults, it is likely that the original image capture willspan a wider field of view than will be generally needed. A givenelectronic imaging device 100 can be equipped with multiple cameras 120,having for example, both lower and higher positioning. The cameras 120will probably be oriented to capture a wide field of view image inportrait mode. Additionally, these cameras 120 can also have pan, tilt,and zoom controls. Thus, it is likely that images captured during thewell being image capture process 550 will capture body imagery of muchmore than just the face 25.

Privacy is a complex issue, which involves personal preferences, familydynamics, legal boundaries, along with societal, cultural, and religiousexpectations. In the case of physiological monitoring system 300, theseprivacy issues interact with legitimate health and medical concerns. Asone example, while there are legitimate reasons to image family members(the subjects 10) using this system, and while this system is typicallyused to image family members and not visitors, other steps can be takenrelative to visitors. For example, when the system identifies anindividual as a non-subject of interest, it can, in addition to notretaining any image data, issue an alert so the non-subject individualknows the system is present. As another option, the local users 10 canshut the physiological monitoring system 300 down for a length of timein advance of an event, such as the hosting of a party.

Considering again the likelihood of an initial wide field image captureby cameras 120, an individual user 10 may be uneasy about having thephysiological monitoring system 300 regularly capture and store torso orfull body images, even though the physiological monitoring system 300 isinstalled in their residence and under their control. As such, thedefault condition for physiological monitoring system 300 is for it toonly save the captured wide field of view images cropped down to justhead and shoulders images, as with the reference image 365 depicted inFIG. 7. Likewise, an alternate default condition can be for thephysiological monitoring system 300 to tabulate and retain wellnessparameter data 410 calculated only from these facial images. However, anindividual user 10 can have legitimate reasons to want to collect imagebased physiological data of themselves or another family member that caninclude torso or full body data. It is likely then that differentdegrees of privacy management will be available, and can even be applieddifferently for different family members. As a result, a privacysettings database 440 is anticipated, including a set of subjectspecific privacy setting parameters (see Table 1).

For example, as an intermediate setting to collecting and maintaining acollection of torso or full body images over time, the physiologicalmonitoring system 300 can collect and retain facial imagery incombination with wellness parameter data 410 for both the face 25 andtorso 50. As another intermediate setting, the physiological monitoringsystem 300 can collect and maintain torso or full body imagery, alongwith the associated wellness parameters 410, but with the image data forthe generally private body areas respectively blurred or cropped out.Certainly, access to the physiological data, and particularly the imagedata, will likely be quite limited in order to maintain confidentiality.Both password protections and data encryption can be used for thispurpose. Other identification based access control technologies, such asvoice, fingerprint, gesture, or other biometric approaches can be used,either individually or in combination. For example, in the familysetting, only the parents are likely designated as the “lead users”, whothen have direct and privileged access to the data and the systemcontrols.

In general, the privacy settings managed by the privacy setting database440 include a list of subjects 10, data regarding identifying attributesto recognize the subjects 10, and access controls associated with thesubjects 10. The privacy settings further include control information onhow to capture, process, store, alert, report, and share the dataproduced by the system 300. These various privacy settings and linkedidentity information can be accessed via a privacy interface, such as agraphical user interface, that can be accessed by a user 10 (a leaduser). These privacy settings nominally apply to image data andimage-based data, but also to non-image data (such as audio). Theprivacy settings can be applied in various ways to control and enablethe sharing of the data with medical professionals. Obviously, thephysiological monitoring system 300 is intended to work in anenvironment where the local individuals (the users 10) generally desireits features and capabilities. As such, the physiological monitoringsystem 300 must provide a useful balance between providing value, whilemaintaining privacy and reducing any imposed burdens unnecessarily.

It is also recognized that circumstances will occur where a user 10intends to use the physiological monitoring system 300 to deliberatelycollect physiological image and wellness data, facial or otherwise, overa period of time. For example, a user 10 may want the teeth or gums of afamily member to be imaged on an ongoing basis. Likewise, image andwellness data may be wanted with respect to a rash or a slowly changingchronic wound. In such instances, the user 10 will need user access atan electronic imaging device 100 to instruct it to deliberately captureand retain the image data, even if the images include privacy sensitiveareas.

It can be anticipated that the both the semantics database 430 and thewellness parameters database 420 would include data that is directlyentered by users 10 (likely by a lead user). For example, users 10 canprovide the semantic data relating to known familial genetic conditionsor historical tendencies, as well as medications being taken, acuteconditions of concern, or other factors. A user 10 can alert or tag thesystem with current wellness data (for example, that a family memberdoes not feel well, is sick, has the measles, or is depressed) that thesystem might not otherwise have knowledge of. Likewise, a user 10 canprovide input to the system regarding the user's own inferences orconclusions concerning the wellness of a family member, whether drawnfrom the output data provided by the system, or from other sources.Generally, the system 300 would then provide appropriate wellnessparameters 410 associated with this information. Additionally, eitherthe system 300 or the users 10 can provide customized wellnessparameters 410 specific to one or more subjects. Any data for userentered wellness parameters 410 can be managed within a subset portionof the wellness parameters database 420. A user's inferences orconclusions can also be input into the inference engine 400, with arequest that it test their potential validity. The dashed lines of FIG.5 c indicate the potential for outside (such as from users 10) inputsand requests to the operational flow of the system.

In a broader context then, the definition of unobtrusive forphysiological monitoring system 300 is somewhat malleable. In general,on an ongoing daily (weekly or monthly) basis, the system is intended tooperate inconspicuously, with little attention or care required. Someusers 10 can be more comfortable with the physiological monitoringsystem 300 if it provides a modest alert, such as a flashing light or anaudio tone, when it is about to proceed with a well-being image capture.Other users 10 may prefer to deliberately direct the system to capturean image, and then deliberately pose for the capture event. Of course,when the physiological monitoring system 300 detects a potential issue,which may reflect a gradual or abrupt physiological change, it issupposed to provide one or more alerts, which can be communicateddifferently depending on the circumstances. For example, an electronicimaging device 100 can provide an alert locally, via a display 110 (withan icon, for example) or a flashing light, an audio tone, but thedetailed data may only be accessible at another location (to a leaduser) on the network 360. Also, various settings regarding privacy,access, and reporting can be user controlled either locally at theelectronic imaging device 100 or by a user interface program run througha computer located within the residence.

In that context it should be understood that the computer 340 whichinterfaces with memory 345, cameras 120, display 110, controller 33, andimage processor 320 can be a specialized device intended for thispurpose, and thus can be separate from any home computer found in aresidence. Such a computer could be part of the imaging deviceassociated with the system or associated with another multi-functionaldevice owned by the user (examples might include a mobile device such asa smart phone or a gaming console). Computer 340 can have its owndisplay and keyboard for direct usage. Alternately computer 340 can beconnected via a network 360 with another computer located in aresidence. Computer 340 can also be an existing computer within aresidence, or a combination of computers, both within and outside aresidence. Likewise memory 345 can include one or more devices, usingone or more data storage technologies, and exist both locally andremotely (off premises).

It is recognized that physiological monitoring system 300, which isprimarily intended for use in the residential environment, must readilyadapt to a wide range of conditions and situations. The ambient lightingcan change quickly over a wide range of conditions. The users can changethe privacy settings, or turn the system off for some duration (such asfor a party). As another dynamic, it can be anticipated that occasionswill occur where the system detects multiple individuals within itsfield of view. As one simple default response, the system can suspendimage capture activities until there is only one potential subjectwithin its field of view. Alternately, the system can analyze theinitial imagery with the subject identification process 510 anddetermine if any of the individuals present are known target subjects.If so, the well-being image capture process 550 can then proceed. Thusit is recognized that both the subject identification process 500 andthe well-being image capture process 550 will need to be capable ofprocessing image data for multiple individuals (users 10)simultaneously.

On another point, physiological monitoring system 300 can revisit theauto-set-up process 520 over the course of time. For example, as aperson ages, particularly during youth, adolescence, or the elderlyyears, dramatic physical changes are typical to the human experience.For example, as adolescents mature, dramatic changes in height andfacial features often occur. Thus while the original reference data(such as a reference image 365) collected at some earlier time likelystill have value, the system 300 will require new baseline data toincrease its utility. Thus, based on an automatic assessment criteria oruser input, the physiological monitoring system 300 can undertake asecondary set-up process to establish new baseline data. Most simply,this likely means that over a suitable period of time the physiologicalmonitoring system 300 will collect and process extra data to use inassembling a new reference data set. For example, inference engine 400can be used to conclude that new baseline data is needed. Inferenceengine 400 can apply knowledge of factors such as the age of subjects10, taken from the semantics database 430 or the wellness parametersdatabase 420, as input to such a decision. A user 10 can also initiate asecondary auto-setup process 520 and provide supporting data.

The physiological image-based data capture has generally been describedas occurring on a daily basis. It is noted that the physiologicalmonitoring system 300 could become quickly become burdened with largeamounts of data. Thus more generally, the physiological monitoringsystem 300 can collect data on a less frequent basis, for example, oncea week, or bi-weekly, monthly, or even quarterly. For example, thesystem can look to capture an image daily, and then compare the resultsto both the reference data and recent data. If the system does notdetect any significant changes, it can retain only wellness parameters410, or the trend line data, or perhaps no data at all. Alternately, ifthe physiological monitoring system 300 detects a change, it can changethe target rate of data measurement and retention for the relevantparameters, for example, from bi-weekly to daily. Additionally, a user10 can change the data capture and retention rate for one or more familymembers.

The physiological monitoring system 300 of the present invention hasbeen generally discussed with respect to the system being used tomonitor a range of wellness conditions, such as nutrition, weight, orposture. As shown in Table 2, there is a wide range of wellness, health,and medical conditions that can potentially be monitored using thissystem, using one or more wellness parameters 410 (see Table 1). Ofcourse, as many physiological changes can be attributed to more than onepotential cause, the problem of false or misdirected positive results isnot insignificant. Although the inference engine 400 can access healthor medical databases 460 in order to screen data and provide informativealerts, physiological monitoring system 300 is best considered to be adiagnostic aid. Considering Table 2, skin color changes and localized ormottled skin color changes can be detected if camera 120 has colorspecific detection devices. For example, cameras 120 can have an imagesensor array 124 (commonly a CCD or CMOS) including sensing pixelsoverlaid with a patterned color filter array (often RGB). To someextent, the native resolution of the camera 120 will impact the abilityto image some of the conditions on Table 2. Simply equipping a camera120 with a zoom lens will enhance the localized resolution and enablebetter imaging of the sclera 32, or skin 40 or skin texture, and otherfeatures.

TABLE 2 Examples of Visual Markers and Associated PhysiologicalConditions Skin color Blue: Lack of oxygen supply in the blood caused bypoor circulation, eating disorder, drug side effects Paleness: Anemia,leukemia, heart conditions Yellow: blood diseases or jaundice (whites ofthe eyes turn yellow as well) Flushing/redness: Stress, anxiety, guilt,exercise, rosacea, heart conditions, exposure to heat and cold Mottledskin exposure to heat and cold, rheumatoid arthritis, dermatitis (skininfection) early changes for some systemic diseases, such as meningitisand encephalitis Localized area skin Rashes - Allergies, diabetes,osteoarthritis discolorations Skin infections - cellulites,endocarditis, dermatitis Bags under eyes Lack of sleep, sinus infection,large intestine (colon) problem, or kidney problems Skin textureLocalized bumps or lesions - rashes and dermatitis, skin infections(impetigo, dermatitis), tuberculosis Response to skin treatments andcosmetics Other (oily, leathery, shiny): many other conditions Skinmoisture level Dry: dehydration, dermatitis, vitamin deficiencies,thyroid disorders, anorexia Moist: Acne and the treatment thereof;General: skin assessment for application of cosmetics and moisturizers,oiliness, pH Hair loss Iron deficiency anemia, thyroid disorders,eczema, lupus, chemotherapy Eyes Bloodshot: fatigue, eyestrain,allergies, infection Eye movement: nystagmus, Parkinson's disease Dryeyes: rheumatoid arthritis, rosacea, common cold Changes in scleracolor: Osteogenesis Imperfecta, Pyknodysostosis, Marfan Syndrome, andothers Nose bleed hypertension Gums - oral health Color changes orulcers: diabetes, cardiovascular diseases, HIV Aids, oral infections,anemia, allergic reactions, Crohn's disease Teeth - oral healthCleanliness - dental care Tongue (color, texture, and size) Anemia,early cancer detection, and other conditions Bulging neck veins heartdisorders Involuntary movements neurological conditions, hand tremors,response to drugs Personal care memory loss, alcoholism, depression,monitoring medical adherence (testing or use of drugs) Gestaltdepression, Alzheimers, tiredness, nourishment, weight gain or loss,posture, gait

However, the imaging of skin texture and structure can be significantlyenhanced if polarization photography is used. According to the paper“Polarized Light Examination and Photography of the Skin”, by R. R.Anderson, in the Archives of Dermatology, Vol. 127, pp. 1000-1005, 1991,the use of polarized illumination light combined with a polarizationanalyzer located prior to the detector, can improve skin imaging. Inparticular, if skin is illuminated with light polarized by a linearpolarizer, and the polarization analyzer is aligned with an orientationparallel to the polarizer, imaging of the skin surface is enhanced.Whereas, if the two polarizers are crossed or orthogonal to each other,skin surface imaging is suppressed and imaging of sub-surface skinstructures is emphasized. As skin texture is principally about surfacequalities, but also depends upon sub-surface structures, bothpolarization-imaging modes can be useful here. Polarization sensitiveimaging can also help with imaging the skin moisture level. In the caseof physiological imaging system 300, this can mean that an electronicimaging device 100 is equipped with a polarized illumination lightsource 215 for illuminating a user. It is noted that providingelectronic imaging device 100 with an illumination light source 215having a known spectrum can also help improve the color correction andnormalization process, as there may be less spectral variation toaccommodate. Although providing such an illumination light source 215can be more obtrusive under some circumstances than using the existingroom lighting or daylight, it is also not unusual for a bathroommedicine cabinet or vanity to be equipped with additional lighting.

The electronic imaging device 100, including cameras 120, can beequipped in other ways to improve and expand the health imagingcapabilities. For example, one or more cameras 120 can have non-visibleinfrared or UV imaging (wherein infrared imaging is done above about 700nm and UV imaging is done below about 400 nm) or multi-spectral imagingcapabilities, which can assist in imaging non-visible changes in theskin, hair, eyes, mouth, saliva, sweat, tears and sub-surface changes inskin structure, skin temperature and skin temperature distributions.More generally, the physiological monitoring system can image light fromthe EM spectrum, nominally in the spectral range from 350-2000 nm orhigher. Other conditions which are reflected in involuntary movements,for example of the eyes 30 or head, might be detected if thephysiological monitoring system 300 captured video images and usedmotion detection algorithms to quantify the motion. As another example,the inference engine 400 can use algorithms to assess an individual'sgestalt (overall health) or emotional state by identifying and trackingpatterns of facial expressions or personal care (grooming andappearance). In particular, facial recognition algorithms can beextended to identify an individual's emotional state by quantifying avariety of facial expressions including but not limited to mouth shapepatterns (for smiles and frowns).

On another note, relative to the previously described image capture andnormalization process, imaging of the sclera 32, and deriving the colorthereof, was described as an approach for color correction which isapplicable to the present invention. Image normalization based onscleral imaging is particularly attractive because of the simplicity andgeneral repeatability. However, the scleral whiteness or color canchange, particularly in a temporary way, when the eyes become“bloodshot” or reddened. Less commonly, some medications, such assteroids, or anemia or aging can cause the sclera 32 to thin and take ona blue tint. There are also various rare medical conditions, such asMarfan's syndrome, which can cause scleral color changes. For suchreasons, it is desirable for the physiological monitoring system 300 toat least occasionally utilize other white-point and color measurementand normalization techniques. By tracking the pre-normalized scleralcolor (as a wellness parameter 410), capture conditions, normalizationtransforms and the associated normalization confidence values the systemcan determine if the color of sclera has actually changed over time.Alternatively a normalized secondary reference feature color can beevaluated to determine if it is within expected ranges. If not it can beinferred that the color of the sclera has changed.

In general, the discussion of physiological imaging system 300 hasfocused on obtaining and analyzing direct front view images of the face25, and maybe the torso 50, of one or more subjects. However, side viewsimages, particularly of the torso 50 are useful as well, particularlyrelative to assessing body weight and posture. These algorithms canaccess the semantic database 430, as well as the wellness and healthdatabases 460 to utilize body type data and health quality data toprovide an assessment. Also, an illumination light source 215 can employstructured lighting, to project a pattern of horizontal or vertical barsof illumination light 210 onto a subject 10 to assist the process ofestimating weight, posture, and gait. This illumination light 210 caninclude light from the infrared spectrum (for example, 830 nm light,which humans cannot see) to maintain the unobtrusiveness relative tosubjects 10.

It should also be understood that the physiological imaging system 300can accept other data inputs for assessing physiological conditions,besides cameras 120. For example, the system can be equipped withsecondary detectors 144 or connections (wireless or physical wires) toother bio-monitoring devices, such as microphones, thermometers, digitalscales, sensing textiles, ear bud sensors, pulse oximeters, and salivatesting devices. One new technology, an electric potential probe, whichis described in application U.S. Patent Application Publication No.2006/0058694 by Clark et al., is particularly attractive as a secondarydetector 144, as it is another non-contact device (like cameras andmicrophones) that can enable a broad range of physiologicalmeasurements. This emerging technology is further described in Electricpotential probes-new directions in the remote sensing of the human body,by C. J. Harland et al., and published in Meas. Sci. Technol. 13 (2002)163-169. Using various secondary detector devices, wellness parameters410 such as blood pressure and heart rate, blood oxygenation, bloodglucose levels, or body temperature, can be measured, and then the datacan be collected, maintained, and tracked within the wellness parametersdatabase 420. Thus its should be understood that system 300 can supportthe sensing and analysis of a wide range of biophysical, biochemical, oremotional markers. Accordingly, while the image normalization process500 and the inference engine 400 have been described principally withrespect to their handling of image-based data, they can be extended toutilize various types of non-image based data, including particularlyvoice data.

Although the physiological monitoring system 300 has been described,relative to FIG. 3, as a networked system, the system had beenpredominately described as including an electronic imaging device 100built into a bathroom vanity. Application in that environment can imposeparticular limitations. For example, the ability of the electronicimaging device 100 to capture images can be impaired if the mirror 136is fogged by condensation as might occur when an individual takes ashower. Of course, the outer, mirrored surface can be heated or coatedto reduce this problem. It is also recognized that in many bathrooms,medicine cabinets are provided behind the mirror. Certainly, it can beexpected that any cameras 120, displays 100, or other detectors orsensors (136, 142, 144) will be competing for at least some of thispotential space. Again considering FIG. 3, electronic imaging devices100 for physiological monitoring system 300 can be positioned elsewherewithin a residence, including behind pictures or bedroom vanities. Asanother example, one or more cameras 120 can be positioned at a computeror a television, much like how web-cameras are used today. In suchcases, the physiological monitoring system 300 can observe other factorsthan the wellness parameters 410 previously described. For example, thephysiological monitoring system 300 can observe the posture, ergonomics,emotional response, attention span, time spent, and fatigue of a subject10 at a computer or television (see FIG. 1). Such data can be useful inassessing mental stress levels, potential repetitive stress disorderssuch as carpal tunnel syndrome, or mental attention to a task. In thecase of children, assessments of mental attention can be useful relativeto conditions like attention deficit disorder (ADD) or for understandingeducational performance. Additionally, the physiological monitoringsystem 300 can also accept inputs from other biomedical devices,including hand held or wearable sensors. These supplemental devices canalso include PDA or cell phone type devices that have imagingcapabilities.

Thus far, the electronic imaging device 100 has been generally describedas an apparatus with one or more cameras 120 mounted at the top of acomputer or television monitor (per FIG. 1) or with one or more cameras120 imbedded behind a mirror (per FIGS. 2 a and 2 b). There are otherpotential constructions however. As a first alternate example, there aremany prior art video-conferencing systems that constitute a display thatsees, wherein a camera 120 and a display 110 are imbedded behind aflickering screen or around a partially reflecting mirror. For example,as shown in FIG. 8 a, a prior art potential electronic imaging device100, which is described in commonly assigned U.S. Pat. No. 7,042,486,entitled “Image capture and display device” by Manico et al., includes acamera 120 and a projector 180, and a flickering or switching screen132. In this system, a semi-transparent (partially silvered) mirror 134is used as a beamsplitter, so that camera 120 and an image displayprojector 180 share a common optical axis 170 to the switching screen132. A shutter 184 modulates the projector 180 to block light fromreaching the screen during a portion of each frame time corresponding toan image capture by camera 120. The shutter 184 is synchronized with theswitching screen 132, such that the shutter's light transmitting statecorresponds to the diffusing state of the switching screen 132, and theimage provide by projector 180 is displayed at switching screen 132.Whereas, the shutter's opaque position corresponds to the lighttransmitting state of switching screen 132. In that case camera 120peers through the switching screen 132 at a subject 10.

In traditional video-conferencing applications, eye contact with minimalparallax error is of utmost importance. Although eye contact can beuseful in certain applications of the present invention, it is not anecessary feature. The traditional configurations for eye contactteleconferencing systems are described in a number of patents, includingthe above Manico '486 patent, and U.S. Pat. Nos. 5,639,151 entitled“Pass-through reflective projection display” and 5,777,665 entitled“Image blocking teleconferencing eye contact terminal”, both toMcNelley, et al.; and U.S. Pat. No. 5,194,955 entitled “Video telephone”to Yoneta et al., for example. As illustrated by the configuration ofFIG. 8 a, these traditional video-conferencing systems, which areburdened with partially silvered mirrors and beam splitters, aretypically bulky, particularly in the depth direction. Additionally, thecurrently commercialized products using this general construction aretargeted to the corporate executive market rather than consumer markets.

As an alternative approach for providing a display that sees, variousparties have proposed a closer integration of image display and sensingcomponents. As one example, illustrated in FIG. 8 b, and described incommonly assigned U.S. patent application Ser. No. 11/555,822, by Kurtzet al., and entitled “An Integrated Display Having Multiple CaptureDevices”, an electronic imaging device 100 with pixel integrated imagedisplay and image capture is shown. This device basically includes acamera 120 that peers through a display 110 that includes aconfiguration of partially transparent pixels. In particular, electronicimaging device 100 includes display pixels 150 and window elements 154formed on a substrate 158, with patterned thin film electroniccomponents 162 providing control signals and drive current. A pattern ofreflective electrodes 168 and transparent electrodes 166 can be used tobring signals and power to each pixel. Some pixels (the window pixels orelements 154) have transparent electrodes 166 both top and bottom, whilethe normal display pixels 154 have reflective electrodes 168 on thebottom side. The display and window pixels can be white light emitters,or color specific (red, green and blue) light emitters, fabricated withfor example, organic light emitting diode (OLED) or polymer lightemitting diode (PLED) technologies. An ensemble of partially transparentpixels (window elements 154) is used to form one or more apertures A,that a camera 120 sees through. Although there are potential imageartifacts, such as a screen door effect, that can effect the capturedimage quality, camera 120 can generally focus and function in a normalway. As the camera 120 is closer to the front display surface, thisconfiguration is much more compact than that of FIG. 8 a, and is closerstructurally to the electronic imaging device 100 shown in FIGS. 2 a and2 b. This integrated approach can be particularly useful in the casethat the display 110 functions as computer or television monitor, as isgenerally depicted in FIG. 1. The FIG. 8 b approach can basically enablea compact integrated electronic imaging device 100, in which the camera120 is embedded behind the display 110, rather than positioned off to aside. Thus, improved eye contact imaging is obtained, which can enhancethe scleral-based image normalization process, as well as some wellnessparameter measurements and assessments.

The present invention has been previously described as typicallyincluding an electronic imaging device 100 which includes a camera 120and a display 110, various sensors, controller 330, and a computer 340and memory 345, which together support a variety of databases, imageprocessing electronics 320, and inference engine 400. Considering theprivacy issues, it can be anticipated that some consumers would preferthat the inference engine 400 and these various databases, including thesemantics database 430, the privacy settings database 440, and thewellness parameters database 440, would exist within their residence.However, it can also be anticipated that some consumers would want thesedata periodically stored in a secure site off their own premises, whichcan be accomplished by a secure data transfer through the network 360(which can include the Internet) to an entity specializing in retainingsuch data.

In a broader context, the assignee of the present invention couldoperate a business in which the physiological monitoring systems 300,assembled with a variety of OEM and assignee provided components, aresold directly, or indirectly, to consumers (users 10). Various thirdparty providers can be enabled to utilize or interface with the systemson behalf of the consumers. For example, data on new or currentprescriptions for medications can be passed from a pharmacy orphysician's office, to the semantics database 430 of a system owned by agiven consumer. A third party entity can manage the secure data transferand interface from the pharmacy to the system of a given user 10, andprovide any related wellness parameters 410. Likewise, external partiescan provide other wellness or set-up data, including wellness parameters410, to one or more user's systems. A third party can also provideadditional analysis modules or extensions for the inference engine 400,which a user 10 can choose to purchase. Alternately, a third party canoperate an enhanced inference engine 400 which the user 10 can agree tohave their data uploaded to, so as to access enhanced analysiscapabilities. For example, the local (at the user's site) inferenceengine 400 might only complete an initial trend or change analysis. Aremote inference engine 400 can then perform a more complete analysis,with interaction with a third party health database 460. Thus, a thirdparty can also maintain and update health databases 460. Again, a givenuser 10 can potentially choose to obtain such health databases from athird party provider, for local use on the system 300 of the given user10. Alternately, the given user can have their data uploaded to a thirdparty to gain the benefits of an enhanced health database 460 maintainedby the third party. Although the physiological monitoring system 300 isintended to enable the capture and analysis of well-being and healthdata, it is not intended to necessarily be a diagnostic system. As oneapproach to diagnostic results, third party providers can also enablemedical diagnosis to the consumers, either locally (through extensionsto the system) or remotely, depending on the complexities of thediagnostic analysis, associated legalities, and the acceptance of theconsumers. Third party participants can also provide health relatedadvertising to the users via this system. In summary, the physiologicalmonitoring system 300 includes a combination of elements, such as acomputer, data storage, software, databases, and perhaps a server, whichcan reside locally (at the residence), or at a remote party, in variouscombinations, depending on the business models of the entities involved,as well as the preferences of the consumers (users 10) involved.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention. As one example, the present invention forphysiological monitoring system 300 has been described as employing astructure of various databases (wellness database 420, semanticsdatabase 430, privacy settings database 440, capture parameters database450. and health database 460). Although the physiological monitoringsystem 300 needs to manage data for wellness, privacy, semantics, imagecapture, and health, it is noted that other approaches and combinationsfor data management beside the above-described databases can be used.Likewise, physiological monitoring system 300 has been described asemploying various processes for image capture, image capture qualityspecification, image normalization, image data assessment, and imagedata reporting. However, other equivalent methods, and changes in theorder thereof, can be used to accomplish the goals of the presentinvention. The physiological monitoring system 300 has also beendescribed relative to its application in residential environments, butthe system can be used in the described form, or a modified form, inother environments for related applications. For example, the system, orvariants thereof, can be used in physician's offices and clinics,long-term care facilities, educational facilities, corporate offices, orfor drug-testing and behavioral therapeutics applications. Theapplication of physiological monitoring system 300 can also be extendedfor the use in monitoring the well-being of family pets. It should beunderstood that the various drawing and figures provided within thisinvention disclosure are intended to be illustrative of the inventionand are not to-scale engineering drawings.

PARTS LIST

-   -   A Aperture    -   10 User (or subject)    -   20 human body    -   25 face    -   30 eye    -   32 sclera    -   34 iris    -   36 pupil    -   40 skin    -   50 torso    -   60 mouth    -   65 hair    -   100 electronic imaging device    -   110 display    -   120 camera    -   122 imaging lens    -   124 sensor array    -   126 spectral filter    -   132 switching screen    -   134 semi-transparent mirror    -   136 mirror    -   140 ambient light detector    -   142 motion detector    -   144 secondary detector    -   150 display pixels    -   154 window elements    -   158 substrate    -   162 thin film electronic components    -   166 transparent electrode

Parts List Con'td

-   -   168 reflective electrode    -   170 optical axis    -   180 projector    -   184 shutter    -   200 ambient light    -   210 illumination light    -   215 illumination light source    -   220 capture light    -   230 display light    -   300 physiological monitoring system    -   310 Image capture system    -   320 image processing electronics    -   330 system controller    -   340 computer    -   345 data storage (or memory)    -   350 alert signal    -   355 communications controller    -   360 network    -   365 reference image    -   367 reference feature    -   370 visual journal    -   400 inference engine    -   402 a, 402 b, 402 c, 402 d steps (for wellness inference or        assessment)    -   410 wellness parameters    -   415 capture parameters    -   420 wellness parameters database    -   430 semantics database    -   440 privacy settings database    -   450 capture parameters database    -   460 health database

Parts List Con'td

-   -   500 image normalization process    -   502 a, 502 b, 502 c steps (for the normalization process)    -   510 subject identification process    -   512 sense an individual step    -   515 user tracking process    -   520 auto-setup process    -   522 a, 522 b, 522 c, 522 d, 522 e, 522 f steps (for auto-setup        process)    -   540 initial imaging process    -   550 well-being image capture process    -   552 a, 552 b, 552 c steps (for well-being data capture)

1. A method for unobtrusive screening the well-being of a person using aplurality of sensing devices in a physiological monitoring system,comprising: operating a physiological monitoring system in a fixedcapture area, using the sensing devices, including at least one camerato capture images without control of ambient light exposure conditions;automatically detecting the presence of a person during an image captureevent; identifying the person as a subject being monitored by the systemand tracking the position and pose of the subject person; associatingknown semantic data with the subject person; using the sensing devicesto automatically and unobtrusively sense one or more body parameters ofthe subject person from the image capture data obtained during the imagecapture events, while automatically determining the capture conditions,including the ambient light exposure and subject pose, as present duringthe capture events; automatically analyzing the captured images and thecapture condition data for a capture event to determine theacceptability of the captured images relative to defined capturecriteria; continuing image capture in an attempt to acquire acceptableimages, until either such images are acquired or the subject personleaves the capture area; calculating and storing wellness parameters,which are derived from the sensed body parameter data, including theimage capture data, which have been determined acceptable relative tocapture conditions; and compiling a wellness record for the subjectperson and automatically evaluating the well-being of the subject personbased on current and prior semantic data including comparing thewellness parameters to previously determined wellness parameters forthat person.
 2. The method according to claim 1 where the definedcapture criteria are defined by acceptable ranges of capture variation.3. The method according to claim 2 wherein the predefined capturecriteria are determined by user privacy settings, user semantics data,the well-being parameters, and the capture parameters, eitherindividually or in combination.
 4. The method according to claim 1wherein the semantic data includes personal data concerning physicalcharacteristics, physiological conditions information of the person orthe person's family, personal behavior, or activity records.
 5. Themethod according to claim 1 including automatically capturing an imageof the person on an ongoing basis, without need for intervention orpreparation by the person.
 6. The method according to claim 1 includingautomatically capturing an image of a person by the camera on an ongoingbasis, when initiated by the person, such that no further interventionor preparation is required of the person after initiation of the imagecapture.
 7. The method according to claim 1 wherein the step ofcalculating wellness parameters includes normalizing the capturecondition data or the body parameter data.
 8. The method according toclaim 1, wherein the sensing step includes providing a sensing devicethat is in physical contact with the person.
 9. The method according toclaim 1 wherein a sensing device is a microphone, which is used todetect voice and other audio data from the person.
 10. The methodaccording to claim 1, wherein sensing of one or more body parametersincludes sensing biophysical markers, biochemical markers or emotionalmarkers.
 11. The method according to claim 10 wherein sensingbiophysical markers includes sensing skin color, skin texture, moistureof skin, face, eyes, hair, neck, shoulders, mouth, hands, posture, oreye movement.
 12. The method according to claim 10, wherein sensingemotional markers includes sensing expression or energy level.
 13. Animage-based system for monitoring physiological conditions of anindividual on an ongoing basis, comprising; an image capture device,provided in a fixed location, which can automatically capture image dataof at least one individual during a capture event; a user identificationmeans which uses captured image data to recognize an individual as asubject being monitored for the capture event, thereby distinguishing agiven subject person from non-subject persons or other subject persons;providing known semantic data associated with the subject person; awell-being image capture structure which quantifies the captureconditions present during the capture event, including the ambient lightexposure conditions and the subject pose, and which screens the capturedimage data relative to predetermined image capture condition criteria toidentify acceptable images; an image normalization structure whichcorrects the captured image data for acceptable images to compensate forvariations in the image capture conditions, to provide normalized imagedata; record keeping means for compiling wellness records for all theknow subjects; a well-being assessment structure which analyzes thenormalized image data relative to physiological conditions of thesubject individual, or changes thereof, based upon the captureconditions and the semantic data of the subject individual, using datafrom the record keeping structure; and output means for providing imagedata or well-being assessments of the individual.
 14. The systemaccording to claim 13, including an inference engine for derivingwellness information.
 15. The system according to claim 14 wherein theinference engine employs additional analysis components, such asanalytical algorithms, which are remotely supplied to the localphysiological monitoring system
 16. The system as in claim 14 includinga second inference engine that is located remotely to the physiologicalassessment system local to an individual.
 17. The system as in claim 16wherein the second inference engine initiates the local physiologicalassessment system to monitor the well-being of an individual relative toa new physiological condition or parameter.
 18. The system according toclaim 13 including means for remotely providing additional wellnessparameter data or physiological condition data to the physiologicalassessment system.
 19. The system according to claim 13 includes meansfor remotely storing wellness parameter data or physiological conditionsdata, for subject persons, from the physiological assessment system. 20.The system according to claim 13, wherein the image capture deviceincludes; a digital still camera, a video camera, or a camera with zoomfeatures.
 21. The system according to claim 13 wherein the image capturedevice captures images in the visible portion of the optical spectrum,the ultraviolet portion of the optical spectrum or the infrared portionof the optical spectrum, or combinations thereof.
 22. The systemaccording to claim 13, which further comprises means for providingillumination of the individual, the illumination being structured,polarized, or from the non-visible portions of the spectrum.
 23. Thesystem according to claim 13, further including one or more additionalsensors for measuring physiological data of the individual.
 24. Thesystem according to claim 13, wherein the output means provides an alertsignal including a visual alert or an audio alert.
 25. The systemaccording to claim 13, wherein the system utilizes one or more referenceimages of an individual.
 26. The system according to claim 13, whereinthe system utilizes a network, which supports system communications.